2024-05-08T19:45:09.946 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-08T19:45:09.946 DEBUG:teuthology.run:Teuthology command: teuthology --name pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7697965 --verbose --description fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} -- /home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7697965/orig.config.yaml 2024-05-08T19:45:09.979 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-08T19:45:10.056 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7697965 branch: wip-pdonnell-testing-20240508.150423-reef description: fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} email: pdonnell@redhat.com first_in_suite: false job_id: '7697965' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-pdonnell-testing-20240508.150423-reef ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: d21e4f961318b56f0d915caa2490c3711bf13c34 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d21e4f961318b56f0d915caa2490c3711bf13c34 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-pdonnell-testing-20240508.150423-reef sha1: d21e4f961318b56f0d915caa2490c3711bf13c34 owner: scheduled_pdonnell@teuthology priority: 65 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - host.b - osd.4 - osd.5 - osd.6 - osd.7 - client.1 seed: 2738 sha1: d21e4f961318b56f0d915caa2490c3711bf13c34 sleep_before_teardown: 0 subset: 17/512 suite: fs suite_branch: wip-pdonnell-testing-20240508.150423-reef suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d21e4f961318b56f0d915caa2490c3711bf13c34/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d21e4f961318b56f0d915caa2490c3711bf13c34 targets: smithi083.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJgb8F6CV4m/d5ijOcU5dhYnLEu0Q18lw+FjDVvH2SzTEIwTAZV0HbGUFYSli213C2g8mMffIZTX35raG92BSmk= smithi188.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBElJptYOGemRB7bbEFahixJe3VC4IWU2Wlp4u/Rn36hT8+LVe+xslvY5+s3KuNTmpyF0NCkmSj1pCyCACnswGIw= tasks: - install: null - cephadm: roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create foo - ceph fs volume create bar - fs.ready: timeout: 300 - ceph-fuse: client.0: cephfs_name: foo - ceph-fuse: client.1: cephfs_name: bar - workunit: clients: all: - suites/dbench.sh - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - sudo sed -i 's/runtime = "runc"/#runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/#runtime = "crun"/runtime = "crun"/g' /usr/share/containers/containers.conf teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-08_19:22:21 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-08T19:45:10.056 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d21e4f961318b56f0d915caa2490c3711bf13c34/qa; will attempt to use it 2024-05-08T19:45:10.057 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d21e4f961318b56f0d915caa2490c3711bf13c34/qa/tasks 2024-05-08T19:45:10.057 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-08T19:45:10.060 INFO:teuthology.task.internal:Checking packages... 2024-05-08T19:45:10.089 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd21e4f961318b56f0d915caa2490c3711bf13c34' 2024-05-08T19:45:10.089 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-08T19:45:10.089 INFO:teuthology.packaging:ref: None 2024-05-08T19:45:10.089 INFO:teuthology.packaging:tag: None 2024-05-08T19:45:10.089 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240508.150423-reef 2024-05-08T19:45:10.090 INFO:teuthology.packaging:sha1: d21e4f961318b56f0d915caa2490c3711bf13c34 2024-05-08T19:45:10.090 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20240508.150423-reef 2024-05-08T19:45:10.237 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1322.gd21e4f96 2024-05-08T19:45:10.239 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-08T19:45:10.246 INFO:teuthology.task.internal:no buildpackages task found 2024-05-08T19:45:10.246 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-08T19:45:10.252 INFO:teuthology.task.internal:Saving configuration 2024-05-08T19:45:10.263 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-08T19:45:10.269 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-08T19:45:10.294 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7697965', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-08 19:40:01.899500', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJgb8F6CV4m/d5ijOcU5dhYnLEu0Q18lw+FjDVvH2SzTEIwTAZV0HbGUFYSli213C2g8mMffIZTX35raG92BSmk='} 2024-05-08T19:45:10.313 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7697965', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-08 19:40:01.901004', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBElJptYOGemRB7bbEFahixJe3VC4IWU2Wlp4u/Rn36hT8+LVe+xslvY5+s3KuNTmpyF0NCkmSj1pCyCACnswGIw='} 2024-05-08T19:45:10.314 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-08T19:45:10.320 INFO:teuthology.task.internal:roles: ubuntu@smithi083.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-05-08T19:45:10.320 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-05-08T19:45:10.320 INFO:teuthology.run_tasks:Running task console_log... 2024-05-08T19:45:10.397 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8f3e53d1f0>, signals=[15]) 2024-05-08T19:45:10.398 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-08T19:45:10.404 INFO:teuthology.task.internal:Opening connections... 2024-05-08T19:45:10.404 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi083.front.sepia.ceph.com 2024-05-08T19:45:10.406 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T19:45:10.494 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi188.front.sepia.ceph.com 2024-05-08T19:45:10.495 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T19:45:10.564 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-08T19:45:10.571 DEBUG:teuthology.orchestra.run.smithi083:> uname -m 2024-05-08T19:45:10.593 INFO:teuthology.orchestra.run.smithi083.stdout:x86_64 2024-05-08T19:45:10.593 DEBUG:teuthology.orchestra.run.smithi083:> cat /etc/os-release 2024-05-08T19:45:10.654 INFO:teuthology.orchestra.run.smithi083.stdout:NAME="CentOS Stream" 2024-05-08T19:45:10.654 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION="8" 2024-05-08T19:45:10.654 INFO:teuthology.orchestra.run.smithi083.stdout:ID="centos" 2024-05-08T19:45:10.654 INFO:teuthology.orchestra.run.smithi083.stdout:ID_LIKE="rhel fedora" 2024-05-08T19:45:10.654 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_ID="8" 2024-05-08T19:45:10.654 INFO:teuthology.orchestra.run.smithi083.stdout:PLATFORM_ID="platform:el8" 2024-05-08T19:45:10.654 INFO:teuthology.orchestra.run.smithi083.stdout:PRETTY_NAME="CentOS Stream 8" 2024-05-08T19:45:10.654 INFO:teuthology.orchestra.run.smithi083.stdout:ANSI_COLOR="0;31" 2024-05-08T19:45:10.654 INFO:teuthology.orchestra.run.smithi083.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-05-08T19:45:10.654 INFO:teuthology.orchestra.run.smithi083.stdout:HOME_URL="https://centos.org/" 2024-05-08T19:45:10.655 INFO:teuthology.orchestra.run.smithi083.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-08T19:45:10.655 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-05-08T19:45:10.655 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-08T19:45:10.655 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server 2024-05-08T19:45:10.676 DEBUG:teuthology.orchestra.run.smithi188:> uname -m 2024-05-08T19:45:10.698 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2024-05-08T19:45:10.698 DEBUG:teuthology.orchestra.run.smithi188:> cat /etc/os-release 2024-05-08T19:45:10.758 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="CentOS Stream" 2024-05-08T19:45:10.759 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="8" 2024-05-08T19:45:10.759 INFO:teuthology.orchestra.run.smithi188.stdout:ID="centos" 2024-05-08T19:45:10.759 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="rhel fedora" 2024-05-08T19:45:10.759 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="8" 2024-05-08T19:45:10.759 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el8" 2024-05-08T19:45:10.759 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="CentOS Stream 8" 2024-05-08T19:45:10.759 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2024-05-08T19:45:10.759 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-05-08T19:45:10.759 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://centos.org/" 2024-05-08T19:45:10.759 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-08T19:45:10.759 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-05-08T19:45:10.759 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-08T19:45:10.760 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2024-05-08T19:45:10.776 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-08T19:45:10.784 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-08T19:45:10.789 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-08T19:45:10.789 DEBUG:teuthology.orchestra.run.smithi083:> test '!' -e /home/ubuntu/cephtest 2024-05-08T19:45:10.791 DEBUG:teuthology.orchestra.run.smithi188:> test '!' -e /home/ubuntu/cephtest 2024-05-08T19:45:10.818 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-08T19:45:10.824 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-08T19:45:10.824 DEBUG:teuthology.orchestra.run.smithi083:> test -z $(ls -A /var/lib/ceph) 2024-05-08T19:45:10.852 DEBUG:teuthology.orchestra.run.smithi188:> test -z $(ls -A /var/lib/ceph) 2024-05-08T19:45:10.883 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-08T19:45:10.935 INFO:teuthology.run_tasks:Running task kernel... 2024-05-08T19:45:10.954 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-08T19:45:10.954 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-08T19:45:10.955 DEBUG:teuthology.orchestra.run.smithi083:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-08T19:45:10.955 DEBUG:teuthology.orchestra.run.smithi188:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-08T19:45:10.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T19:45:10.975 DEBUG:teuthology.orchestra.run.smithi188:> uname -r 2024-05-08T19:45:10.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T19:45:10.976 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-05-08T19:45:11.034 INFO:teuthology.orchestra.run.smithi188.stdout:4.18.0-489.el8.x86_64 2024-05-08T19:45:11.034 INFO:teuthology.task.kernel:Running kernel on smithi188: 4.18.0-489.el8.x86_64 2024-05-08T19:45:11.034 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum install -y kernel 2024-05-08T19:45:11.038 INFO:teuthology.orchestra.run.smithi083.stdout:4.18.0-489.el8.x86_64 2024-05-08T19:45:11.038 INFO:teuthology.task.kernel:Running kernel on smithi083: 4.18.0-489.el8.x86_64 2024-05-08T19:45:11.038 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum install -y kernel 2024-05-08T19:45:15.672 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 8 - AppStream 13 MB/s | 28 MB 00:02 2024-05-08T19:45:15.834 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - AppStream 13 MB/s | 28 MB 00:02 2024-05-08T19:45:22.027 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - BaseOS 8.5 MB/s | 10 MB 00:01 2024-05-08T19:45:22.639 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 8 - BaseOS 5.4 MB/s | 10 MB 00:01 2024-05-08T19:45:24.043 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - Extras 45 kB/s | 18 kB 00:00 2024-05-08T19:45:24.360 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - Extras common packages 37 kB/s | 7.7 kB 00:00 2024-05-08T19:45:24.731 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 8 - Extras 39 kB/s | 18 kB 00:00 2024-05-08T19:45:24.803 INFO:teuthology.orchestra.run.smithi188.stdout:Copr repo for python3-asyncssh owned by ceph 9.8 kB/s | 3.5 kB 00:00 2024-05-08T19:45:25.215 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 8 - Extras common packages 21 kB/s | 7.7 kB 00:00 2024-05-08T19:45:25.458 INFO:teuthology.orchestra.run.smithi083.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-05-08T19:45:33.425 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 2.1 MB/s | 16 MB 00:07 2024-05-08T19:45:37.234 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 315 kB/s | 24 kB 00:00 2024-05-08T19:45:40.347 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-05-08T19:45:40.347 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-05-08T19:45:40.481 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:45:40.482 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:45:40.482 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-05-08T19:45:40.482 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:45:40.482 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-05-08T19:45:40.482 INFO:teuthology.orchestra.run.smithi083.stdout: kernel x86_64 4.18.0-552.1.1.el8 baseos 10 M 2024-05-08T19:45:40.482 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-05-08T19:45:40.482 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-core x86_64 4.18.0-552.1.1.el8 baseos 44 M 2024-05-08T19:45:40.483 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules x86_64 4.18.0-552.1.1.el8 baseos 36 M 2024-05-08T19:45:40.483 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:45:40.483 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-05-08T19:45:40.483 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:45:40.483 INFO:teuthology.orchestra.run.smithi083.stdout:Install 3 Packages 2024-05-08T19:45:40.483 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:45:40.483 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 90 M 2024-05-08T19:45:40.484 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 96 M 2024-05-08T19:45:40.484 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-05-08T19:45:42.993 INFO:teuthology.orchestra.run.smithi083.stdout:(1/3): kernel-4.18.0-552.1.1.el8.x86_64.rpm 5.4 MB/s | 10 MB 00:01 2024-05-08T19:45:44.543 INFO:teuthology.orchestra.run.smithi083.stdout:(2/3): kernel-modules-4.18.0-552.1.1.el8.x86_64 11 MB/s | 36 MB 00:03 2024-05-08T19:45:44.961 INFO:teuthology.orchestra.run.smithi083.stdout:(3/3): kernel-core-4.18.0-552.1.1.el8.x86_64.rp 11 MB/s | 44 MB 00:03 2024-05-08T19:45:44.961 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:45:44.961 INFO:teuthology.orchestra.run.smithi083.stdout:Total 20 MB/s | 90 MB 00:04 2024-05-08T19:45:45.645 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-05-08T19:45:46.065 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-05-08T19:45:46.065 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-05-08T19:45:47.316 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-05-08T19:45:47.339 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-05-08T19:45:50.141 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-05-08T19:45:50.685 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-core-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-08T19:45:51.782 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 621 kB/s | 16 MB 00:26 2024-05-08T19:45:54.244 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-core-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-08T19:45:54.626 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-modules-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-08T19:45:55.615 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 358 kB/s | 24 kB 00:00 2024-05-08T19:45:58.063 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-08T19:45:58.341 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-08T19:45:58.754 INFO:teuthology.orchestra.run.smithi188.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-05-08T19:45:58.754 INFO:teuthology.orchestra.run.smithi188.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-05-08T19:45:58.887 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:45:58.888 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:45:58.888 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-05-08T19:45:58.888 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:45:58.888 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-05-08T19:45:58.888 INFO:teuthology.orchestra.run.smithi188.stdout: kernel x86_64 4.18.0-552.1.1.el8 baseos 10 M 2024-05-08T19:45:58.889 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-05-08T19:45:58.889 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-core x86_64 4.18.0-552.1.1.el8 baseos 44 M 2024-05-08T19:45:58.889 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-modules x86_64 4.18.0-552.1.1.el8 baseos 36 M 2024-05-08T19:45:58.889 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:45:58.889 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-05-08T19:45:58.889 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:45:58.889 INFO:teuthology.orchestra.run.smithi188.stdout:Install 3 Packages 2024-05-08T19:45:58.889 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:45:58.890 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 90 M 2024-05-08T19:45:58.890 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 96 M 2024-05-08T19:45:58.890 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-05-08T19:46:01.358 INFO:teuthology.orchestra.run.smithi188.stdout:(1/3): kernel-4.18.0-552.1.1.el8.x86_64.rpm 4.3 MB/s | 10 MB 00:02 2024-05-08T19:46:09.765 INFO:teuthology.orchestra.run.smithi188.stdout:(2/3): kernel-modules-4.18.0-552.1.1.el8.x86_64 3.4 MB/s | 36 MB 00:10 2024-05-08T19:46:11.973 INFO:teuthology.orchestra.run.smithi188.stdout:(3/3): kernel-core-4.18.0-552.1.1.el8.x86_64.rp 3.4 MB/s | 44 MB 00:13 2024-05-08T19:46:11.973 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:46:11.973 INFO:teuthology.orchestra.run.smithi188.stdout:Total 6.9 MB/s | 90 MB 00:13 2024-05-08T19:46:12.666 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-05-08T19:46:13.079 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-05-08T19:46:13.079 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-05-08T19:46:14.310 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-05-08T19:46:14.337 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-05-08T19:46:17.130 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-05-08T19:46:17.638 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-core-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-08T19:46:20.012 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-core-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-08T19:46:20.037 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-08T19:46:20.390 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-08T19:46:20.390 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-08T19:46:20.390 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-core-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-08T19:46:20.853 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-modules-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-08T19:46:20.853 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:46:20.853 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-05-08T19:46:20.854 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-4.18.0-552.1.1.el8.x86_64 2024-05-08T19:46:20.854 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-core-4.18.0-552.1.1.el8.x86_64 2024-05-08T19:46:20.854 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-4.18.0-552.1.1.el8.x86_64 2024-05-08T19:46:20.854 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:46:20.854 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:46:21.010 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel | sort -rV | head -n 1 2024-05-08T19:46:21.215 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-core-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-08T19:46:21.230 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-4.18.0-552.1.1.el8.x86_64 2024-05-08T19:46:21.230 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.1.1.el8.x86_64 2024-05-08T19:46:21.231 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.1.1.el8.x86_64 2024-05-08T19:46:21.231 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-08T19:46:21.231 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-08T19:46:21.231 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-05-08T19:46:21.231 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi083.front.sepia.ceph.com, path=None, version=distro) 2024-05-08T19:46:21.232 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel | sort -rV | head -n 1 2024-05-08T19:46:21.442 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-4.18.0-552.1.1.el8.x86_64 2024-05-08T19:46:21.443 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.1.1.el8.x86_64 2024-05-08T19:46:21.443 DEBUG:teuthology.orchestra.run.smithi083:> sudo rpm -qi grub2-tools 2024-05-08T19:46:21.476 INFO:teuthology.orchestra.run.smithi083.stdout:Name : grub2-tools 2024-05-08T19:46:21.476 INFO:teuthology.orchestra.run.smithi083.stdout:Epoch : 1 2024-05-08T19:46:21.476 INFO:teuthology.orchestra.run.smithi083.stdout:Version : 2.02 2024-05-08T19:46:21.476 INFO:teuthology.orchestra.run.smithi083.stdout:Release : 129.el8 2024-05-08T19:46:21.476 INFO:teuthology.orchestra.run.smithi083.stdout:Architecture: x86_64 2024-05-08T19:46:21.476 INFO:teuthology.orchestra.run.smithi083.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-05-08T19:46:21.477 INFO:teuthology.orchestra.run.smithi083.stdout:Group : System Environment/Base 2024-05-08T19:46:21.477 INFO:teuthology.orchestra.run.smithi083.stdout:Size : 9265586 2024-05-08T19:46:21.477 INFO:teuthology.orchestra.run.smithi083.stdout:License : GPLv3+ 2024-05-08T19:46:21.477 INFO:teuthology.orchestra.run.smithi083.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-05-08T19:46:21.477 INFO:teuthology.orchestra.run.smithi083.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-05-08T19:46:21.477 INFO:teuthology.orchestra.run.smithi083.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-05-08T19:46:21.477 INFO:teuthology.orchestra.run.smithi083.stdout:Build Host : kbuilder.bsys.centos.org 2024-05-08T19:46:21.477 INFO:teuthology.orchestra.run.smithi083.stdout:Relocations : (not relocatable) 2024-05-08T19:46:21.477 INFO:teuthology.orchestra.run.smithi083.stdout:Packager : CentOS BuildSystem 2024-05-08T19:46:21.477 INFO:teuthology.orchestra.run.smithi083.stdout:Vendor : CentOS 2024-05-08T19:46:21.477 INFO:teuthology.orchestra.run.smithi083.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-08T19:46:21.477 INFO:teuthology.orchestra.run.smithi083.stdout:Summary : Support tools for GRUB. 2024-05-08T19:46:21.477 INFO:teuthology.orchestra.run.smithi083.stdout:Description : 2024-05-08T19:46:21.477 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:46:21.478 INFO:teuthology.orchestra.run.smithi083.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-08T19:46:21.478 INFO:teuthology.orchestra.run.smithi083.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-08T19:46:21.478 INFO:teuthology.orchestra.run.smithi083.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-08T19:46:21.478 INFO:teuthology.orchestra.run.smithi083.stdout:hardware devices. 2024-05-08T19:46:21.478 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:46:21.478 INFO:teuthology.orchestra.run.smithi083.stdout:This subpackage provides tools for support of all platforms. 2024-05-08T19:46:21.479 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-08T19:46:21.479 INFO:teuthology.task.kernel:Updating grub on smithi083 to boot 4.18.0-552.1.1.el8.x86_64 2024-05-08T19:46:21.479 DEBUG:teuthology.orchestra.run.smithi083:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-08T19:46:21.603 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-modules-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-08T19:46:23.301 INFO:teuthology.orchestra.run.smithi083.stderr:Generating grub configuration file ... 2024-05-08T19:46:25.047 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-modules-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-08T19:46:25.335 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-08T19:46:25.469 INFO:teuthology.orchestra.run.smithi083.stderr:done 2024-05-08T19:46:25.470 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2024-05-08T19:46:25.503 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.Xt7iy30KP7 2024-05-08T19:46:25.503 DEBUG:teuthology.orchestra.run.smithi083:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Xt7iy30KP7 2024-05-08T19:46:25.575 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.Xt7iy30KP7 2024-05-08T19:46:25.719 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.Xt7iy30KP7 is 6KB 2024-05-08T19:46:25.766 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.Xt7iy30KP7 2024-05-08T19:46:25.784 DEBUG:teuthology.orchestra.run.smithi083:> sudo /bin/ls /boot/loader/entries || true 2024-05-08T19:46:25.856 INFO:teuthology.orchestra.run.smithi083.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-05-08T19:46:25.856 INFO:teuthology.orchestra.run.smithi083.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-05-08T19:46:25.856 INFO:teuthology.orchestra.run.smithi083.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-05-08T19:46:25.856 INFO:teuthology.orchestra.run.smithi083.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.1.1.el8.x86_64.conf 2024-05-08T19:46:25.857 DEBUG:teuthology.orchestra.run.smithi083:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.1.1.el8.x86_64 2024-05-08T19:46:26.018 DEBUG:teuthology.orchestra.run.smithi083:> sudo shutdown -r now 2024-05-08T19:46:47.175 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-core-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-08T19:46:47.197 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-modules-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-08T19:46:47.547 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-08T19:46:47.548 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-08T19:46:47.548 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-core-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-08T19:46:48.048 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-modules-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-08T19:46:48.049 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:46:48.049 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-05-08T19:46:48.049 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-4.18.0-552.1.1.el8.x86_64 2024-05-08T19:46:48.049 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-core-4.18.0-552.1.1.el8.x86_64 2024-05-08T19:46:48.049 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-modules-4.18.0-552.1.1.el8.x86_64 2024-05-08T19:46:48.049 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:46:48.049 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:46:48.194 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q kernel | sort -rV | head -n 1 2024-05-08T19:46:48.408 INFO:teuthology.orchestra.run.smithi188.stdout:kernel-4.18.0-552.1.1.el8.x86_64 2024-05-08T19:46:48.408 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.1.1.el8.x86_64 2024-05-08T19:46:48.409 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.1.1.el8.x86_64 2024-05-08T19:46:48.409 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-08T19:46:48.409 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-08T19:46:48.409 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-05-08T19:46:48.409 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi188.front.sepia.ceph.com, path=None, version=distro) 2024-05-08T19:46:48.409 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q kernel | sort -rV | head -n 1 2024-05-08T19:46:48.621 INFO:teuthology.orchestra.run.smithi188.stdout:kernel-4.18.0-552.1.1.el8.x86_64 2024-05-08T19:46:48.621 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.1.1.el8.x86_64 2024-05-08T19:46:48.622 DEBUG:teuthology.orchestra.run.smithi188:> sudo rpm -qi grub2-tools 2024-05-08T19:46:48.655 INFO:teuthology.orchestra.run.smithi188.stdout:Name : grub2-tools 2024-05-08T19:46:48.655 INFO:teuthology.orchestra.run.smithi188.stdout:Epoch : 1 2024-05-08T19:46:48.655 INFO:teuthology.orchestra.run.smithi188.stdout:Version : 2.02 2024-05-08T19:46:48.655 INFO:teuthology.orchestra.run.smithi188.stdout:Release : 129.el8 2024-05-08T19:46:48.655 INFO:teuthology.orchestra.run.smithi188.stdout:Architecture: x86_64 2024-05-08T19:46:48.655 INFO:teuthology.orchestra.run.smithi188.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-05-08T19:46:48.655 INFO:teuthology.orchestra.run.smithi188.stdout:Group : System Environment/Base 2024-05-08T19:46:48.656 INFO:teuthology.orchestra.run.smithi188.stdout:Size : 9265586 2024-05-08T19:46:48.656 INFO:teuthology.orchestra.run.smithi188.stdout:License : GPLv3+ 2024-05-08T19:46:48.656 INFO:teuthology.orchestra.run.smithi188.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-05-08T19:46:48.656 INFO:teuthology.orchestra.run.smithi188.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-05-08T19:46:48.656 INFO:teuthology.orchestra.run.smithi188.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-05-08T19:46:48.656 INFO:teuthology.orchestra.run.smithi188.stdout:Build Host : kbuilder.bsys.centos.org 2024-05-08T19:46:48.656 INFO:teuthology.orchestra.run.smithi188.stdout:Relocations : (not relocatable) 2024-05-08T19:46:48.656 INFO:teuthology.orchestra.run.smithi188.stdout:Packager : CentOS BuildSystem 2024-05-08T19:46:48.656 INFO:teuthology.orchestra.run.smithi188.stdout:Vendor : CentOS 2024-05-08T19:46:48.656 INFO:teuthology.orchestra.run.smithi188.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-08T19:46:48.656 INFO:teuthology.orchestra.run.smithi188.stdout:Summary : Support tools for GRUB. 2024-05-08T19:46:48.656 INFO:teuthology.orchestra.run.smithi188.stdout:Description : 2024-05-08T19:46:48.656 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:46:48.657 INFO:teuthology.orchestra.run.smithi188.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-08T19:46:48.657 INFO:teuthology.orchestra.run.smithi188.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-08T19:46:48.657 INFO:teuthology.orchestra.run.smithi188.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-08T19:46:48.657 INFO:teuthology.orchestra.run.smithi188.stdout:hardware devices. 2024-05-08T19:46:48.657 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:46:48.657 INFO:teuthology.orchestra.run.smithi188.stdout:This subpackage provides tools for support of all platforms. 2024-05-08T19:46:48.658 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-08T19:46:48.658 INFO:teuthology.task.kernel:Updating grub on smithi188 to boot 4.18.0-552.1.1.el8.x86_64 2024-05-08T19:46:48.658 DEBUG:teuthology.orchestra.run.smithi188:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-08T19:46:50.371 INFO:teuthology.orchestra.run.smithi188.stderr:Generating grub configuration file ... 2024-05-08T19:46:52.473 INFO:teuthology.orchestra.run.smithi188.stderr:done 2024-05-08T19:46:52.474 DEBUG:teuthology.orchestra.run.smithi188:> mktemp 2024-05-08T19:46:52.507 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.6i5la1pvhp 2024-05-08T19:46:52.507 DEBUG:teuthology.orchestra.run.smithi188:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6i5la1pvhp 2024-05-08T19:46:52.581 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.6i5la1pvhp 2024-05-08T19:46:52.721 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.6i5la1pvhp is 6KB 2024-05-08T19:46:52.769 DEBUG:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.6i5la1pvhp 2024-05-08T19:46:52.787 DEBUG:teuthology.orchestra.run.smithi188:> sudo /bin/ls /boot/loader/entries || true 2024-05-08T19:46:52.859 INFO:teuthology.orchestra.run.smithi188.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-05-08T19:46:52.859 INFO:teuthology.orchestra.run.smithi188.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-05-08T19:46:52.859 INFO:teuthology.orchestra.run.smithi188.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-05-08T19:46:52.859 INFO:teuthology.orchestra.run.smithi188.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.1.1.el8.x86_64.conf 2024-05-08T19:46:52.860 DEBUG:teuthology.orchestra.run.smithi188:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.1.1.el8.x86_64 2024-05-08T19:46:53.000 DEBUG:teuthology.orchestra.run.smithi188:> sudo shutdown -r now 2024-05-08T19:46:56.023 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-05-08T19:46:56.024 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-05-08T19:46:56.024 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T19:47:14.591 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-05-08T19:47:23.010 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-05-08T19:47:23.011 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-05-08T19:47:23.011 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T19:47:23.594 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-05-08T19:47:23.594 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T19:47:26.658 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-05-08T19:47:38.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-05-08T19:47:38.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T19:47:41.470 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.188 2024-05-08T19:47:50.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-05-08T19:47:50.479 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T19:47:53.534 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.188 2024-05-08T19:48:05.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-05-08T19:48:05.547 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T19:48:10.894 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-05-08T19:48:12.122 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-05-08T19:48:12.122 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-552.1.1.el8.x86_64"... 2024-05-08T19:48:12.122 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-05-08T19:48:12.396 INFO:teuthology.orchestra.run.smithi083.stdout:4.18.0-552.1.1.el8.x86_64 2024-05-08T19:48:12.397 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.1.1.el8.x86_64 vs 4.18.0-552.1.1.el8.x86_64 2024-05-08T19:48:12.397 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-08T19:48:12.397 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-08T19:48:13.398 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-05-08T19:48:13.399 DEBUG:teuthology.orchestra.run.smithi083:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-08T19:48:13.459 INFO:teuthology.orchestra.run.smithi083.stdout:ttyS1 2024-05-08T19:48:13.478 DEBUG:teuthology.parallel:result is None 2024-05-08T19:48:38.004 DEBUG:teuthology.orchestra.run.smithi188:> true 2024-05-08T19:48:39.348 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-05-08T19:48:39.349 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-552.1.1.el8.x86_64"... 2024-05-08T19:48:39.349 DEBUG:teuthology.orchestra.run.smithi188:> uname -r 2024-05-08T19:48:39.365 INFO:teuthology.orchestra.run.smithi188.stdout:4.18.0-552.1.1.el8.x86_64 2024-05-08T19:48:39.365 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.1.1.el8.x86_64 vs 4.18.0-552.1.1.el8.x86_64 2024-05-08T19:48:39.365 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-08T19:48:39.365 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-08T19:48:40.366 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-05-08T19:48:40.367 DEBUG:teuthology.orchestra.run.smithi188:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-08T19:48:40.426 INFO:teuthology.orchestra.run.smithi188.stdout:ttyS1 2024-05-08T19:48:40.443 DEBUG:teuthology.parallel:result is None 2024-05-08T19:48:40.443 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-08T19:48:40.450 INFO:teuthology.task.internal:Creating test directory... 2024-05-08T19:48:40.450 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-08T19:48:40.453 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-08T19:48:40.473 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-08T19:48:40.480 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-08T19:48:40.489 INFO:teuthology.task.internal:Creating archive directory... 2024-05-08T19:48:40.489 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-08T19:48:40.515 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-08T19:48:40.548 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-08T19:48:40.556 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-08T19:48:40.556 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-08T19:48:40.582 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-08T19:48:40.617 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T19:48:40.621 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T19:48:40.628 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T19:48:40.631 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T19:48:40.632 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-08T19:48:40.640 INFO:teuthology.task.internal:Configuring sudo... 2024-05-08T19:48:40.640 DEBUG:teuthology.orchestra.run.smithi083:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-08T19:48:40.672 DEBUG:teuthology.orchestra.run.smithi188:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-08T19:48:40.701 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-08T19:48:40.710 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-08T19:48:40.710 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-08T19:48:40.743 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-08T19:48:40.761 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-08T19:48:40.849 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-08T19:48:40.897 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-08T19:48:40.983 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-08T19:48:41.027 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-05-08T19:48:41.027 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-08T19:48:41.095 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-08T19:48:41.138 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-08T19:48:41.202 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-08T19:48:41.246 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-08T19:48:41.287 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-05-08T19:48:41.287 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-08T19:48:41.354 DEBUG:teuthology.orchestra.run.smithi083:> sudo service rsyslog restart 2024-05-08T19:48:41.356 DEBUG:teuthology.orchestra.run.smithi188:> sudo service rsyslog restart 2024-05-08T19:48:41.433 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-08T19:48:41.445 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-08T19:48:41.696 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-08T19:48:41.704 INFO:teuthology.task.internal:Starting timer... 2024-05-08T19:48:41.704 INFO:teuthology.run_tasks:Running task pcp... 2024-05-08T19:48:41.716 INFO:teuthology.run_tasks:Running task selinux... 2024-05-08T19:48:41.724 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-05-08T19:48:41.725 DEBUG:teuthology.orchestra.run.smithi083:> sudo service auditd rotate 2024-05-08T19:48:41.874 INFO:teuthology.orchestra.run.smithi083.stdout:Rotating logs: 2024-05-08T19:48:41.876 DEBUG:teuthology.orchestra.run.smithi188:> sudo service auditd rotate 2024-05-08T19:48:41.962 INFO:teuthology.orchestra.run.smithi188.stdout:Rotating logs: 2024-05-08T19:48:41.963 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-08T19:48:41.964 DEBUG:teuthology.orchestra.run.smithi083:> /usr/sbin/getenforce 2024-05-08T19:48:42.003 INFO:teuthology.orchestra.run.smithi083.stdout:Permissive 2024-05-08T19:48:42.004 DEBUG:teuthology.orchestra.run.smithi188:> /usr/sbin/getenforce 2024-05-08T19:48:42.026 INFO:teuthology.orchestra.run.smithi188.stdout:Permissive 2024-05-08T19:48:42.026 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi083.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi188.front.sepia.ceph.com': 'permissive'} 2024-05-08T19:48:42.027 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-08T19:48:42.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T19:48:42.073 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-08T19:48:42.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T19:48:42.100 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-08T19:48:42.100 DEBUG:teuthology.orchestra.run.smithi083:> sudo /usr/sbin/setenforce permissive 2024-05-08T19:48:42.141 DEBUG:teuthology.orchestra.run.smithi188:> sudo /usr/sbin/setenforce permissive 2024-05-08T19:48:42.185 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-08T19:48:42.195 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-05-08T19:48:42.200 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-05-08T19:48:42.327 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-08T19:48:42.343 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-08T19:48:42.345 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi083.front.sepia.ceph.com,smithi188.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-08T19:54:38.553 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi083.front.sepia.ceph.com'), Remote(name='ubuntu@smithi188.front.sepia.ceph.com')] 2024-05-08T19:54:38.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-05-08T19:54:38.556 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T19:54:38.639 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-05-08T19:54:38.709 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-05-08T19:54:38.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-05-08T19:54:38.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T19:54:38.791 DEBUG:teuthology.orchestra.run.smithi188:> true 2024-05-08T19:54:38.861 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-05-08T19:54:38.861 INFO:teuthology.run_tasks:Running task clock... 2024-05-08T19:54:38.871 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-08T19:54:38.871 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-08T19:54:38.871 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-08T19:54:38.874 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-08T19:54:38.874 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-08T19:54:38.905 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-08T19:54:38.919 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-08T19:54:38.934 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-08T19:54:38.948 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-08T19:54:38.987 INFO:teuthology.orchestra.run.smithi083.stderr:sudo: ntpd: command not found 2024-05-08T19:54:38.997 INFO:teuthology.orchestra.run.smithi083.stdout:506 Cannot talk to daemon 2024-05-08T19:54:39.011 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-08T19:54:39.019 INFO:teuthology.orchestra.run.smithi188.stderr:sudo: ntpd: command not found 2024-05-08T19:54:39.023 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-08T19:54:39.030 INFO:teuthology.orchestra.run.smithi188.stdout:506 Cannot talk to daemon 2024-05-08T19:54:39.043 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-08T19:54:39.056 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-08T19:54:39.097 INFO:teuthology.orchestra.run.smithi083.stderr:bash: ntpq: command not found 2024-05-08T19:54:39.099 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-08T19:54:39.100 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-05-08T19:54:39.100 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T19:54:39.100 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T19:54:39.100 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T19:54:39.100 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T19:54:39.123 INFO:teuthology.orchestra.run.smithi188.stderr:bash: ntpq: command not found 2024-05-08T19:54:39.127 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-08T19:54:39.127 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-05-08T19:54:39.127 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T19:54:39.127 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T19:54:39.127 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T19:54:39.127 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T19:54:39.127 INFO:teuthology.run_tasks:Running task install... 2024-05-08T19:54:39.136 DEBUG:teuthology.task.install:project ceph 2024-05-08T19:54:39.136 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd21e4f961318b56f0d915caa2490c3711bf13c34'}} 2024-05-08T19:54:39.136 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd21e4f961318b56f0d915caa2490c3711bf13c34'} 2024-05-08T19:54:39.136 INFO:teuthology.task.install:Using flavor: default 2024-05-08T19:54:39.142 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-08T19:54:39.142 INFO:teuthology.task.install:extra packages: [] 2024-05-08T19:54:39.142 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd21e4f961318b56f0d915caa2490c3711bf13c34', 'tag': None, 'wait_for_package': False} 2024-05-08T19:54:39.143 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d21e4f961318b56f0d915caa2490c3711bf13c34 2024-05-08T19:54:39.145 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd21e4f961318b56f0d915caa2490c3711bf13c34', 'tag': None, 'wait_for_package': False} 2024-05-08T19:54:39.145 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d21e4f961318b56f0d915caa2490c3711bf13c34 2024-05-08T19:54:39.297 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.150423-reef/d21e4f961318b56f0d915caa2490c3711bf13c34/centos/8/flavors/default/ 2024-05-08T19:54:39.297 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1322.gd21e4f96 2024-05-08T19:54:39.307 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.150423-reef/d21e4f961318b56f0d915caa2490c3711bf13c34/centos/8/flavors/default/ 2024-05-08T19:54:39.307 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1322.gd21e4f96 2024-05-08T19:54:39.590 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.150423-reef/d21e4f961318b56f0d915caa2490c3711bf13c34/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.150423-reef/d21e4f961318b56f0d915caa2490c3711bf13c34/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.150423-reef/d21e4f961318b56f0d915caa2490c3711bf13c34/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-08T19:54:39.591 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-05-08T19:54:39.591 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-08T19:54:39.620 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-05-08T19:54:39.621 DEBUG:teuthology.orchestra.run.smithi083:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d21e4f961318b56f0d915caa2490c3711bf13c34/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-08T19:54:39.664 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.150423-reef/d21e4f961318b56f0d915caa2490c3711bf13c34/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.150423-reef/d21e4f961318b56f0d915caa2490c3711bf13c34/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.150423-reef/d21e4f961318b56f0d915caa2490c3711bf13c34/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-08T19:54:39.665 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-05-08T19:54:39.665 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-08T19:54:39.691 DEBUG:teuthology.orchestra.run.smithi083:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-08T19:54:39.693 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-05-08T19:54:39.693 DEBUG:teuthology.orchestra.run.smithi188:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d21e4f961318b56f0d915caa2490c3711bf13c34/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-08T19:54:39.761 DEBUG:teuthology.orchestra.run.smithi188:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-08T19:54:39.770 DEBUG:teuthology.orchestra.run.smithi083:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-08T19:54:39.837 INFO:teuthology.orchestra.run.smithi083.stdout:check_obsoletes = 1 2024-05-08T19:54:39.838 DEBUG:teuthology.orchestra.run.smithi188:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-08T19:54:39.839 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum clean all 2024-05-08T19:54:39.904 INFO:teuthology.orchestra.run.smithi188.stdout:check_obsoletes = 1 2024-05-08T19:54:39.905 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum clean all 2024-05-08T19:54:40.118 INFO:teuthology.orchestra.run.smithi083.stdout:78 files removed 2024-05-08T19:54:40.148 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-radosgw 2024-05-08T19:54:40.171 INFO:teuthology.orchestra.run.smithi188.stdout:78 files removed 2024-05-08T19:54:40.199 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-radosgw 2024-05-08T19:54:41.309 INFO:teuthology.orchestra.run.smithi083.stdout:ceph packages for x86_64 106 kB/s | 80 kB 00:00 2024-05-08T19:54:41.388 INFO:teuthology.orchestra.run.smithi188.stdout:ceph packages for x86_64 102 kB/s | 80 kB 00:00 2024-05-08T19:54:42.069 INFO:teuthology.orchestra.run.smithi083.stdout:ceph noarch packages 16 kB/s | 11 kB 00:00 2024-05-08T19:54:42.236 INFO:teuthology.orchestra.run.smithi188.stdout:ceph noarch packages 14 kB/s | 11 kB 00:00 2024-05-08T19:54:42.777 INFO:teuthology.orchestra.run.smithi083.stdout:ceph source packages 2.2 kB/s | 1.5 kB 00:00 2024-05-08T19:54:42.972 INFO:teuthology.orchestra.run.smithi188.stdout:ceph source packages 2.1 kB/s | 1.5 kB 00:00 2024-05-08T19:54:44.861 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 8 - AppStream 14 MB/s | 28 MB 00:02 2024-05-08T19:54:45.105 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - AppStream 13 MB/s | 28 MB 00:02 2024-05-08T19:54:50.695 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 8 - BaseOS 13 MB/s | 10 MB 00:00 2024-05-08T19:54:50.905 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - BaseOS 13 MB/s | 10 MB 00:00 2024-05-08T19:54:52.537 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 8 - Extras 72 kB/s | 18 kB 00:00 2024-05-08T19:54:52.855 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 8 - Extras common packages 37 kB/s | 7.7 kB 00:00 2024-05-08T19:54:52.889 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - Extras 48 kB/s | 18 kB 00:00 2024-05-08T19:54:53.348 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 7.7 kB 00:00 2024-05-08T19:54:53.798 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 8 - PowerTools 5.8 MB/s | 5.0 MB 00:00 2024-05-08T19:54:54.416 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - PowerTools 5.1 MB/s | 5.0 MB 00:00 2024-05-08T19:54:54.849 INFO:teuthology.orchestra.run.smithi083.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2024-05-08T19:54:55.476 INFO:teuthology.orchestra.run.smithi188.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-05-08T19:54:57.035 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-05-08T19:55:00.802 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 382 kB/s | 24 kB 00:00 2024-05-08T19:55:04.502 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:55:04.504 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:55:04.504 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-05-08T19:55:04.504 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:55:04.504 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-05-08T19:55:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 13 M 2024-05-08T19:55:04.505 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrading: 2024-05-08T19:55:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: librados2 x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 3.7 M 2024-05-08T19:55:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1 x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 3.6 M 2024-05-08T19:55:04.505 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-05-08T19:55:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 5.5 M 2024-05-08T19:55:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 22 M 2024-05-08T19:55:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 24 k 2024-05-08T19:55:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-05-08T19:55:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2 x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 782 k 2024-05-08T19:55:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-05-08T19:55:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-05-08T19:55:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 474 k 2024-05-08T19:55:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-05-08T19:55:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2 x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 6.5 M 2024-05-08T19:55:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-05-08T19:55:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-05-08T19:55:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 45 k 2024-05-08T19:55:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 129 k 2024-05-08T19:55:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 197 k 2024-05-08T19:55:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-05-08T19:55:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 370 k 2024-05-08T19:55:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 365 k 2024-05-08T19:55:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 107 k 2024-05-08T19:55:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-05-08T19:55:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:55:04.507 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-05-08T19:55:04.507 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:55:04.507 INFO:teuthology.orchestra.run.smithi188.stdout:Install 21 Packages 2024-05-08T19:55:04.507 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrade 2 Packages 2024-05-08T19:55:04.507 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:55:04.507 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 60 M 2024-05-08T19:55:04.507 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-05-08T19:55:07.917 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 1.3 MB/s | 16 MB 00:12 2024-05-08T19:55:08.503 INFO:teuthology.orchestra.run.smithi188.stdout:(1/23): ceph-base-18.2.2-1322.gd21e4f96.el8.x86 2.7 MB/s | 5.5 MB 00:02 2024-05-08T19:55:08.603 INFO:teuthology.orchestra.run.smithi188.stdout:(2/23): ceph-selinux-18.2.2-1322.gd21e4f96.el8. 244 kB/s | 24 kB 00:00 2024-05-08T19:55:08.811 INFO:teuthology.orchestra.run.smithi188.stdout:(3/23): libcephfs2-18.2.2-1322.gd21e4f96.el8.x8 3.7 MB/s | 782 kB 00:00 2024-05-08T19:55:08.953 INFO:teuthology.orchestra.run.smithi188.stdout:(4/23): libradosstriper1-18.2.2-1322.gd21e4f96. 3.3 MB/s | 474 kB 00:00 2024-05-08T19:55:10.220 INFO:teuthology.orchestra.run.smithi188.stdout:(5/23): librgw2-18.2.2-1322.gd21e4f96.el8.x86_6 5.2 MB/s | 6.5 MB 00:01 2024-05-08T19:55:10.346 INFO:teuthology.orchestra.run.smithi188.stdout:(6/23): python3-ceph-argparse-18.2.2-1322.gd21e 357 kB/s | 45 kB 00:00 2024-05-08T19:55:10.454 INFO:teuthology.orchestra.run.smithi188.stdout:(7/23): python3-ceph-common-18.2.2-1322.gd21e4f 1.2 MB/s | 129 kB 00:00 2024-05-08T19:55:10.571 INFO:teuthology.orchestra.run.smithi188.stdout:(8/23): python3-cephfs-18.2.2-1322.gd21e4f96.el 1.6 MB/s | 197 kB 00:00 2024-05-08T19:55:11.046 INFO:teuthology.orchestra.run.smithi188.stdout:(9/23): python3-rados-18.2.2-1322.gd21e4f96.el8 780 kB/s | 370 kB 00:00 2024-05-08T19:55:11.156 INFO:teuthology.orchestra.run.smithi188.stdout:(10/23): python3-rbd-18.2.2-1322.gd21e4f96.el8. 3.3 MB/s | 365 kB 00:00 2024-05-08T19:55:11.214 INFO:teuthology.orchestra.run.smithi188.stdout:(11/23): python3-rgw-18.2.2-1322.gd21e4f96.el8. 1.8 MB/s | 107 kB 00:00 2024-05-08T19:55:11.692 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 403 kB/s | 24 kB 00:00 2024-05-08T19:55:11.739 INFO:teuthology.orchestra.run.smithi188.stdout:(12/23): librdkafka-1.6.1-1.el8.x86_64.rpm 1.3 MB/s | 677 kB 00:00 2024-05-08T19:55:11.839 INFO:teuthology.orchestra.run.smithi188.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.5 MB/s | 259 kB 00:00 2024-05-08T19:55:11.931 INFO:teuthology.orchestra.run.smithi188.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 480 kB/s | 44 kB 00:00 2024-05-08T19:55:12.150 INFO:teuthology.orchestra.run.smithi188.stdout:(15/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 217 kB/s | 47 kB 00:00 2024-05-08T19:55:12.732 INFO:teuthology.orchestra.run.smithi188.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 525 kB/s | 306 kB 00:00 2024-05-08T19:55:12.849 INFO:teuthology.orchestra.run.smithi188.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 504 kB/s | 59 kB 00:00 2024-05-08T19:55:12.974 INFO:teuthology.orchestra.run.smithi188.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 600 kB/s | 75 kB 00:00 2024-05-08T19:55:13.591 INFO:teuthology.orchestra.run.smithi188.stdout:(19/23): thrift-0.13.0-2.el8.x86_64.rpm 2.8 MB/s | 1.7 MB 00:00 2024-05-08T19:55:14.658 INFO:teuthology.orchestra.run.smithi188.stdout:(20/23): librados2-18.2.2-1322.gd21e4f96.el8.x8 3.5 MB/s | 3.7 MB 00:01 2024-05-08T19:55:15.366 INFO:teuthology.orchestra.run.smithi188.stdout:(21/23): librbd1-18.2.2-1322.gd21e4f96.el8.x86_ 5.0 MB/s | 3.6 MB 00:00 2024-05-08T19:55:15.378 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:55:15.381 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:55:15.381 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-05-08T19:55:15.381 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:55:15.381 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-05-08T19:55:15.381 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 13 M 2024-05-08T19:55:15.381 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrading: 2024-05-08T19:55:15.381 INFO:teuthology.orchestra.run.smithi083.stdout: librados2 x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 3.7 M 2024-05-08T19:55:15.382 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1 x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 3.6 M 2024-05-08T19:55:15.382 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-05-08T19:55:15.382 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 5.5 M 2024-05-08T19:55:15.382 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 22 M 2024-05-08T19:55:15.382 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 24 k 2024-05-08T19:55:15.382 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-05-08T19:55:15.382 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2 x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 782 k 2024-05-08T19:55:15.382 INFO:teuthology.orchestra.run.smithi083.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-05-08T19:55:15.382 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-05-08T19:55:15.382 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1 x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 474 k 2024-05-08T19:55:15.382 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-05-08T19:55:15.382 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2 x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 6.5 M 2024-05-08T19:55:15.382 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-05-08T19:55:15.383 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-05-08T19:55:15.383 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 45 k 2024-05-08T19:55:15.383 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 129 k 2024-05-08T19:55:15.383 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 197 k 2024-05-08T19:55:15.383 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-05-08T19:55:15.383 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 370 k 2024-05-08T19:55:15.383 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 365 k 2024-05-08T19:55:15.383 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 107 k 2024-05-08T19:55:15.383 INFO:teuthology.orchestra.run.smithi083.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-05-08T19:55:15.383 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:55:15.383 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-05-08T19:55:15.383 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:55:15.383 INFO:teuthology.orchestra.run.smithi083.stdout:Install 21 Packages 2024-05-08T19:55:15.383 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrade 2 Packages 2024-05-08T19:55:15.384 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:55:15.384 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 60 M 2024-05-08T19:55:15.384 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-05-08T19:55:17.675 INFO:teuthology.orchestra.run.smithi188.stdout:(22/23): ceph-radosgw-18.2.2-1322.gd21e4f96.el8 1.2 MB/s | 13 MB 00:11 2024-05-08T19:55:17.743 INFO:teuthology.orchestra.run.smithi083.stdout:(1/23): ceph-base-18.2.2-1322.gd21e4f96.el8.x86 2.7 MB/s | 5.5 MB 00:02 2024-05-08T19:55:17.851 INFO:teuthology.orchestra.run.smithi083.stdout:(2/23): ceph-selinux-18.2.2-1322.gd21e4f96.el8. 224 kB/s | 24 kB 00:00 2024-05-08T19:55:18.143 INFO:teuthology.orchestra.run.smithi083.stdout:(3/23): libcephfs2-18.2.2-1322.gd21e4f96.el8.x8 2.6 MB/s | 782 kB 00:00 2024-05-08T19:55:18.335 INFO:teuthology.orchestra.run.smithi083.stdout:(4/23): libradosstriper1-18.2.2-1322.gd21e4f96. 2.4 MB/s | 474 kB 00:00 2024-05-08T19:55:20.227 INFO:teuthology.orchestra.run.smithi083.stdout:(5/23): ceph-common-18.2.2-1322.gd21e4f96.el8.x 4.8 MB/s | 22 MB 00:04 2024-05-08T19:55:20.311 INFO:teuthology.orchestra.run.smithi083.stdout:(6/23): librgw2-18.2.2-1322.gd21e4f96.el8.x86_6 3.3 MB/s | 6.5 MB 00:01 2024-05-08T19:55:20.336 INFO:teuthology.orchestra.run.smithi083.stdout:(7/23): python3-ceph-argparse-18.2.2-1322.gd21e 412 kB/s | 45 kB 00:00 2024-05-08T19:55:20.494 INFO:teuthology.orchestra.run.smithi083.stdout:(8/23): ceph-radosgw-18.2.2-1322.gd21e4f96.el8. 2.7 MB/s | 13 MB 00:04 2024-05-08T19:55:20.520 INFO:teuthology.orchestra.run.smithi083.stdout:(9/23): python3-ceph-common-18.2.2-1322.gd21e4f 618 kB/s | 129 kB 00:00 2024-05-08T19:55:20.545 INFO:teuthology.orchestra.run.smithi083.stdout:(10/23): python3-cephfs-18.2.2-1322.gd21e4f96.e 942 kB/s | 197 kB 00:00 2024-05-08T19:55:20.612 INFO:teuthology.orchestra.run.smithi083.stdout:(11/23): python3-rados-18.2.2-1322.gd21e4f96.el 3.1 MB/s | 370 kB 00:00 2024-05-08T19:55:20.637 INFO:teuthology.orchestra.run.smithi083.stdout:(12/23): python3-rgw-18.2.2-1322.gd21e4f96.el8. 1.1 MB/s | 107 kB 00:00 2024-05-08T19:55:20.671 INFO:teuthology.orchestra.run.smithi083.stdout:(13/23): python3-rbd-18.2.2-1322.gd21e4f96.el8. 2.4 MB/s | 365 kB 00:00 2024-05-08T19:55:20.997 INFO:teuthology.orchestra.run.smithi083.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 135 kB/s | 44 kB 00:00 2024-05-08T19:55:21.097 INFO:teuthology.orchestra.run.smithi083.stdout:(15/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 566 kB/s | 259 kB 00:00 2024-05-08T19:55:21.122 INFO:teuthology.orchestra.run.smithi083.stdout:(16/23): librdkafka-1.6.1-1.el8.x86_64.rpm 1.3 MB/s | 677 kB 00:00 2024-05-08T19:55:21.156 INFO:teuthology.orchestra.run.smithi083.stdout:(17/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 298 kB/s | 47 kB 00:00 2024-05-08T19:55:21.258 INFO:teuthology.orchestra.run.smithi188.stdout:(23/23): ceph-common-18.2.2-1322.gd21e4f96.el8. 1.5 MB/s | 22 MB 00:14 2024-05-08T19:55:21.260 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:55:21.260 INFO:teuthology.orchestra.run.smithi188.stdout:Total 3.6 MB/s | 60 MB 00:16 2024-05-08T19:55:21.289 INFO:teuthology.orchestra.run.smithi083.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 362 kB/s | 59 kB 00:00 2024-05-08T19:55:21.302 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-05-08T19:55:21.315 INFO:teuthology.orchestra.run.smithi083.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 489 kB/s | 75 kB 00:00 2024-05-08T19:55:21.348 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-05-08T19:55:21.348 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-05-08T19:55:21.348 INFO:teuthology.orchestra.run.smithi083.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2024-05-08T19:55:21.482 INFO:teuthology.orchestra.run.smithi083.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 9.0 MB/s | 1.7 MB 00:00 2024-05-08T19:55:21.708 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-05-08T19:55:21.711 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-05-08T19:55:22.082 INFO:teuthology.orchestra.run.smithi083.stdout:(22/23): librbd1-18.2.2-1322.gd21e4f96.el8.x86_ 4.9 MB/s | 3.6 MB 00:00 2024-05-08T19:55:22.232 INFO:teuthology.orchestra.run.smithi083.stdout:(23/23): librados2-18.2.2-1322.gd21e4f96.el8.x8 4.0 MB/s | 3.7 MB 00:00 2024-05-08T19:55:22.237 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:55:22.237 INFO:teuthology.orchestra.run.smithi083.stdout:Total 8.7 MB/s | 60 MB 00:06 2024-05-08T19:55:22.278 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-05-08T19:55:22.324 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-05-08T19:55:22.324 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-05-08T19:55:22.701 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-05-08T19:55:22.704 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-05-08T19:55:22.867 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-05-08T19:55:23.102 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2024-05-08T19:55:23.308 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2024-05-08T19:55:23.504 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2024-05-08T19:55:23.899 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-05-08T19:55:23.936 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2024-05-08T19:55:24.122 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2024-05-08T19:55:24.212 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librados2-2:18.2.2-1322.gd21e4f96.el8.x86_64 4/25 2024-05-08T19:55:24.308 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-2:18.2.2-1322.gd21e4f96.el8.x86_64 4/25 2024-05-08T19:55:24.319 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2024-05-08T19:55:24.515 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2024-05-08T19:55:24.602 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rados-2:18.2.2-1322.gd21e4f96.el8.x86_64 5/25 2024-05-08T19:55:24.762 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs2-2:18.2.2-1322.gd21e4f96.el8.x86_64 6/25 2024-05-08T19:55:24.916 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2024-05-08T19:55:25.115 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librados2-2:18.2.2-1322.gd21e4f96.el8.x86_64 4/25 2024-05-08T19:55:25.153 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs2-2:18.2.2-1322.gd21e4f96.el8.x86_64 6/25 2024-05-08T19:55:25.218 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-2:18.2.2-1322.gd21e4f96.el8.x86_64 4/25 2024-05-08T19:55:25.312 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librbd1-2:18.2.2-1322.gd21e4f96.el8.x86_64 7/25 2024-05-08T19:55:25.381 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-2:18.2.2-1322.gd21e4f96.el8.x86_64 7/25 2024-05-08T19:55:25.462 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rados-2:18.2.2-1322.gd21e4f96.el8.x86_64 5/25 2024-05-08T19:55:25.629 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-05-08T19:55:25.682 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs2-2:18.2.2-1322.gd21e4f96.el8.x86_64 6/25 2024-05-08T19:55:25.810 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-05-08T19:55:25.998 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-argparse-2:18.2.2-1322.gd21e4f96.el8. 10/25 2024-05-08T19:55:26.071 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs2-2:18.2.2-1322.gd21e4f96.el8.x86_64 6/25 2024-05-08T19:55:26.172 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cephfs-2:18.2.2-1322.gd21e4f96.el8.x86_64 11/25 2024-05-08T19:55:26.249 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librbd1-2:18.2.2-1322.gd21e4f96.el8.x86_64 7/25 2024-05-08T19:55:26.315 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-2:18.2.2-1322.gd21e4f96.el8.x86_64 7/25 2024-05-08T19:55:26.404 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 12/25 2024-05-08T19:55:26.517 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-05-08T19:55:26.570 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libradosstriper1-2:18.2.2-1322.gd21e4f96.el8.x86_6 13/25 2024-05-08T19:55:26.622 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1322.gd21e4f96.el8.x86_6 13/25 2024-05-08T19:55:26.697 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-05-08T19:55:26.845 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2024-05-08T19:55:26.898 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-argparse-2:18.2.2-1322.gd21e4f96.el8. 10/25 2024-05-08T19:55:27.092 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cephfs-2:18.2.2-1322.gd21e4f96.el8.x86_64 11/25 2024-05-08T19:55:27.291 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 12/25 2024-05-08T19:55:27.473 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libradosstriper1-2:18.2.2-1322.gd21e4f96.el8.x86_6 13/25 2024-05-08T19:55:27.524 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1322.gd21e4f96.el8.x86_6 13/25 2024-05-08T19:55:27.623 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2024-05-08T19:55:27.733 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2024-05-08T19:55:27.786 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librgw2-2:18.2.2-1322.gd21e4f96.el8.x86_64 16/25 2024-05-08T19:55:27.877 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librgw2-2:18.2.2-1322.gd21e4f96.el8.x86_64 16/25 2024-05-08T19:55:28.061 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 17/25 2024-05-08T19:55:28.246 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-05-08T19:55:28.470 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-common-2:18.2.2-1322.gd21e4f96.el8.x8 19/25 2024-05-08T19:55:28.498 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2024-05-08T19:55:28.640 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librgw2-2:18.2.2-1322.gd21e4f96.el8.x86_64 16/25 2024-05-08T19:55:28.711 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librgw2-2:18.2.2-1322.gd21e4f96.el8.x86_64 16/25 2024-05-08T19:55:28.873 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 17/25 2024-05-08T19:55:29.049 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-05-08T19:55:29.232 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-common-2:18.2.2-1322.gd21e4f96.el8.x8 19/25 2024-05-08T19:55:31.020 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 20/25 2024-05-08T19:55:31.244 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 20/25 2024-05-08T19:55:31.790 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 20/25 2024-05-08T19:55:31.866 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 20/25 2024-05-08T19:55:32.006 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 20/25 2024-05-08T19:55:32.136 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-base-2:18.2.2-1322.gd21e4f96.el8.x86_64 21/25 2024-05-08T19:55:32.232 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-base-2:18.2.2-1322.gd21e4f96.el8.x86_64 21/25 2024-05-08T19:55:32.411 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-selinux-2:18.2.2-1322.gd21e4f96.el8.x86_64 22/25 2024-05-08T19:55:32.619 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 20/25 2024-05-08T19:55:32.874 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-base-2:18.2.2-1322.gd21e4f96.el8.x86_64 21/25 2024-05-08T19:55:32.978 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-base-2:18.2.2-1322.gd21e4f96.el8.x86_64 21/25 2024-05-08T19:55:33.186 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-selinux-2:18.2.2-1322.gd21e4f96.el8.x86_64 22/25 2024-05-08T19:55:51.794 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1322.gd21e4f96.el8.x86_64 22/25 2024-05-08T19:55:51.795 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /sys 2024-05-08T19:55:51.795 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /proc 2024-05-08T19:55:51.795 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /dev 2024-05-08T19:55:51.795 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /run 2024-05-08T19:55:51.795 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /mnt 2024-05-08T19:55:51.795 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /var/tmp 2024-05-08T19:55:51.795 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /home 2024-05-08T19:55:51.795 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /tmp 2024-05-08T19:55:51.795 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /dev 2024-05-08T19:55:51.795 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:55:52.356 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1322.gd21e4f96.el8.x86_64 22/25 2024-05-08T19:55:52.356 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /sys 2024-05-08T19:55:52.356 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /proc 2024-05-08T19:55:52.356 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /dev 2024-05-08T19:55:52.356 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /run 2024-05-08T19:55:52.356 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /mnt 2024-05-08T19:55:52.356 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /var/tmp 2024-05-08T19:55:52.356 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /home 2024-05-08T19:55:52.356 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /tmp 2024-05-08T19:55:52.356 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /dev 2024-05-08T19:55:52.356 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:55:53.229 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-radosgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 23/25 2024-05-08T19:55:53.249 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 23/25 2024-05-08T19:55:53.249 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-05-08T19:55:53.457 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-05-08T19:55:53.457 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-05-08T19:55:53.763 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-radosgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 23/25 2024-05-08T19:55:53.784 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 23/25 2024-05-08T19:55:53.784 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-05-08T19:55:53.980 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-05-08T19:55:53.981 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-05-08T19:55:54.150 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-05-08T19:55:54.151 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-base-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/25 2024-05-08T19:55:54.151 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 2/25 2024-05-08T19:55:54.151 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-radosgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 3/25 2024-05-08T19:55:54.151 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-selinux-2:18.2.2-1322.gd21e4f96.el8.x86_64 4/25 2024-05-08T19:55:54.151 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs2-2:18.2.2-1322.gd21e4f96.el8.x86_64 5/25 2024-05-08T19:55:54.151 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libradosstriper1-2:18.2.2-1322.gd21e4f96.el8.x86_6 6/25 2024-05-08T19:55:54.151 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librgw2-2:18.2.2-1322.gd21e4f96.el8.x86_64 7/25 2024-05-08T19:55:54.151 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1322.gd21e4f96.el8. 8/25 2024-05-08T19:55:54.151 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-common-2:18.2.2-1322.gd21e4f96.el8.x8 9/25 2024-05-08T19:55:54.151 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cephfs-2:18.2.2-1322.gd21e4f96.el8.x86_64 10/25 2024-05-08T19:55:54.151 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rados-2:18.2.2-1322.gd21e4f96.el8.x86_64 11/25 2024-05-08T19:55:54.152 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 12/25 2024-05-08T19:55:54.152 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 13/25 2024-05-08T19:55:54.152 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-05-08T19:55:54.152 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-05-08T19:55:54.152 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-05-08T19:55:54.152 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-05-08T19:55:54.152 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-05-08T19:55:54.152 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-05-08T19:55:54.152 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-05-08T19:55:54.152 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-05-08T19:55:54.152 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-2:18.2.2-1322.gd21e4f96.el8.x86_64 22/25 2024-05-08T19:55:54.152 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-05-08T19:55:54.152 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-2:18.2.2-1322.gd21e4f96.el8.x86_64 24/25 2024-05-08T19:55:54.624 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-05-08T19:55:54.624 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-base-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/25 2024-05-08T19:55:54.624 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 2/25 2024-05-08T19:55:54.624 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-radosgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 3/25 2024-05-08T19:55:54.624 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-selinux-2:18.2.2-1322.gd21e4f96.el8.x86_64 4/25 2024-05-08T19:55:54.624 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs2-2:18.2.2-1322.gd21e4f96.el8.x86_64 5/25 2024-05-08T19:55:54.624 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libradosstriper1-2:18.2.2-1322.gd21e4f96.el8.x86_6 6/25 2024-05-08T19:55:54.624 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librgw2-2:18.2.2-1322.gd21e4f96.el8.x86_64 7/25 2024-05-08T19:55:54.624 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1322.gd21e4f96.el8. 8/25 2024-05-08T19:55:54.625 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-common-2:18.2.2-1322.gd21e4f96.el8.x8 9/25 2024-05-08T19:55:54.625 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cephfs-2:18.2.2-1322.gd21e4f96.el8.x86_64 10/25 2024-05-08T19:55:54.625 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rados-2:18.2.2-1322.gd21e4f96.el8.x86_64 11/25 2024-05-08T19:55:54.625 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 12/25 2024-05-08T19:55:54.625 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 13/25 2024-05-08T19:55:54.625 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-05-08T19:55:54.625 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-05-08T19:55:54.625 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-05-08T19:55:54.625 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-05-08T19:55:54.625 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-05-08T19:55:54.625 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-05-08T19:55:54.625 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-05-08T19:55:54.625 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-05-08T19:55:54.625 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-2:18.2.2-1322.gd21e4f96.el8.x86_64 22/25 2024-05-08T19:55:54.625 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-05-08T19:55:54.626 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-2:18.2.2-1322.gd21e4f96.el8.x86_64 24/25 2024-05-08T19:55:55.097 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-05-08T19:55:55.097 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:55:55.097 INFO:teuthology.orchestra.run.smithi188.stdout:Upgraded: 2024-05-08T19:55:55.097 INFO:teuthology.orchestra.run.smithi188.stdout: librados2-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.097 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.097 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-05-08T19:55:55.097 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.097 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.097 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.097 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.098 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-05-08T19:55:55.098 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.098 INFO:teuthology.orchestra.run.smithi188.stdout: liboath-2.6.2-3.el8.x86_64 2024-05-08T19:55:55.098 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-05-08T19:55:55.098 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.098 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-05-08T19:55:55.098 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.098 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind-1.3.1-3.el8.x86_64 2024-05-08T19:55:55.099 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-05-08T19:55:55.099 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.099 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.099 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.099 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-05-08T19:55:55.099 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.099 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.099 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.099 INFO:teuthology.orchestra.run.smithi188.stdout: thrift-0.13.0-2.el8.x86_64 2024-05-08T19:55:55.099 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:55:55.099 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:55:55.373 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-test 2024-05-08T19:55:55.510 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-05-08T19:55:55.510 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:55:55.510 INFO:teuthology.orchestra.run.smithi083.stdout:Upgraded: 2024-05-08T19:55:55.510 INFO:teuthology.orchestra.run.smithi083.stdout: librados2-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.510 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.510 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-05-08T19:55:55.510 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.510 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.510 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.510 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.510 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-05-08T19:55:55.511 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.511 INFO:teuthology.orchestra.run.smithi083.stdout: liboath-2.6.2-3.el8.x86_64 2024-05-08T19:55:55.511 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-05-08T19:55:55.511 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.511 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-05-08T19:55:55.511 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.511 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind-1.3.1-3.el8.x86_64 2024-05-08T19:55:55.511 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-05-08T19:55:55.511 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.511 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.511 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.511 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-05-08T19:55:55.511 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.512 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.512 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:55:55.512 INFO:teuthology.orchestra.run.smithi083.stdout: thrift-0.13.0-2.el8.x86_64 2024-05-08T19:55:55.512 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:55:55.512 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:55:55.791 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-test 2024-05-08T19:55:56.101 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:00:56 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:55:56.514 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:45 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:55:57.746 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:55:57.747 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:55:57.747 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-05-08T19:55:57.747 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:55:57.747 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-05-08T19:55:57.747 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 47 M 2024-05-08T19:55:57.747 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-05-08T19:55:57.747 INFO:teuthology.orchestra.run.smithi188.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-05-08T19:55:57.748 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 142 k 2024-05-08T19:55:57.748 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-05-08T19:55:57.748 INFO:teuthology.orchestra.run.smithi188.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-05-08T19:55:57.748 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-05-08T19:55:57.748 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:55:57.748 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-05-08T19:55:57.748 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:55:57.748 INFO:teuthology.orchestra.run.smithi188.stdout:Install 6 Packages 2024-05-08T19:55:57.748 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:55:57.749 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 48 M 2024-05-08T19:55:57.749 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 213 M 2024-05-08T19:55:57.749 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-05-08T19:55:58.170 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:55:58.171 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:55:58.171 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-05-08T19:55:58.171 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:55:58.171 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-05-08T19:55:58.171 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 47 M 2024-05-08T19:55:58.171 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-05-08T19:55:58.171 INFO:teuthology.orchestra.run.smithi083.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-05-08T19:55:58.171 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 142 k 2024-05-08T19:55:58.171 INFO:teuthology.orchestra.run.smithi083.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-05-08T19:55:58.171 INFO:teuthology.orchestra.run.smithi083.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-05-08T19:55:58.171 INFO:teuthology.orchestra.run.smithi083.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-05-08T19:55:58.171 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:55:58.172 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-05-08T19:55:58.172 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:55:58.172 INFO:teuthology.orchestra.run.smithi083.stdout:Install 6 Packages 2024-05-08T19:55:58.172 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:55:58.172 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 48 M 2024-05-08T19:55:58.173 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 213 M 2024-05-08T19:55:58.173 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-05-08T19:55:58.222 INFO:teuthology.orchestra.run.smithi188.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 518 kB/s | 203 kB 00:00 2024-05-08T19:55:58.306 INFO:teuthology.orchestra.run.smithi188.stdout:(2/6): libcephsqlite-18.2.2-1322.gd21e4f96.el8. 298 kB/s | 142 kB 00:00 2024-05-08T19:55:58.331 INFO:teuthology.orchestra.run.smithi188.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-05-08T19:55:58.439 INFO:teuthology.orchestra.run.smithi188.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 651 kB/s | 71 kB 00:00 2024-05-08T19:55:58.623 INFO:teuthology.orchestra.run.smithi083.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 547 kB/s | 203 kB 00:00 2024-05-08T19:55:58.723 INFO:teuthology.orchestra.run.smithi083.stdout:(2/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-05-08T19:55:58.732 INFO:teuthology.orchestra.run.smithi188.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 759 kB/s | 323 kB 00:00 2024-05-08T19:55:58.773 INFO:teuthology.orchestra.run.smithi083.stdout:(3/6): libcephsqlite-18.2.2-1322.gd21e4f96.el8. 272 kB/s | 142 kB 00:00 2024-05-08T19:55:58.915 INFO:teuthology.orchestra.run.smithi083.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.6 MB/s | 323 kB 00:00 2024-05-08T19:55:59.074 INFO:teuthology.orchestra.run.smithi083.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 235 kB/s | 71 kB 00:00 2024-05-08T19:56:04.714 INFO:teuthology.orchestra.run.smithi188.stdout:(6/6): ceph-test-18.2.2-1322.gd21e4f96.el8.x86_ 6.9 MB/s | 47 MB 00:06 2024-05-08T19:56:04.716 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:56:04.716 INFO:teuthology.orchestra.run.smithi188.stdout:Total 6.9 MB/s | 48 MB 00:06 2024-05-08T19:56:04.757 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-05-08T19:56:04.778 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-05-08T19:56:04.778 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-05-08T19:56:05.088 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-05-08T19:56:05.090 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-05-08T19:56:05.565 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-05-08T19:56:05.798 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-05-08T19:56:05.997 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-05-08T19:56:06.152 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-05-08T19:56:06.235 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-05-08T19:56:06.406 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-05-08T19:56:06.552 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephsqlite-2:18.2.2-1322.gd21e4f96.el8.x86_64 5/6 2024-05-08T19:56:11.100 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1322.gd21e4f96.el8.x86_64 5/6 2024-05-08T19:56:11.350 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-test-2:18.2.2-1322.gd21e4f96.el8.x86_64 6/6 2024-05-08T19:56:11.637 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-test-2:18.2.2-1322.gd21e4f96.el8.x86_64 6/6 2024-05-08T19:56:11.637 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-test-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/6 2024-05-08T19:56:11.638 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephsqlite-2:18.2.2-1322.gd21e4f96.el8.x86_64 2/6 2024-05-08T19:56:11.638 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-05-08T19:56:11.638 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-05-08T19:56:11.638 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-05-08T19:56:11.931 INFO:teuthology.orchestra.run.smithi083.stdout:(6/6): ceph-test-18.2.2-1322.gd21e4f96.el8.x86_ 3.4 MB/s | 47 MB 00:13 2024-05-08T19:56:11.933 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:56:11.933 INFO:teuthology.orchestra.run.smithi083.stdout:Total 3.5 MB/s | 48 MB 00:13 2024-05-08T19:56:11.974 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-05-08T19:56:11.995 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-05-08T19:56:11.995 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-05-08T19:56:12.089 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-05-08T19:56:12.089 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:56:12.089 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-05-08T19:56:12.089 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:56:12.089 INFO:teuthology.orchestra.run.smithi188.stdout: jq-1.6-8.el8.x86_64 2024-05-08T19:56:12.089 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:56:12.089 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-05-08T19:56:12.089 INFO:teuthology.orchestra.run.smithi188.stdout: socat-1.7.4.1-1.el8.x86_64 2024-05-08T19:56:12.090 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-05-08T19:56:12.090 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:56:12.090 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:56:12.300 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph 2024-05-08T19:56:12.316 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-05-08T19:56:12.319 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-05-08T19:56:12.774 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-05-08T19:56:12.983 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-05-08T19:56:13.034 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:13 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:56:13.161 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-05-08T19:56:13.317 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-05-08T19:56:13.393 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-05-08T19:56:13.563 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-05-08T19:56:13.708 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephsqlite-2:18.2.2-1322.gd21e4f96.el8.x86_64 5/6 2024-05-08T19:56:14.703 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:56:14.706 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:56:14.706 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-05-08T19:56:14.706 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:56:14.707 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-05-08T19:56:14.707 INFO:teuthology.orchestra.run.smithi188.stdout: ceph x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 6.4 k 2024-05-08T19:56:14.707 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-05-08T19:56:14.707 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 2.3 M 2024-05-08T19:56:14.707 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 1.6 M 2024-05-08T19:56:14.707 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 245 k 2024-05-08T19:56:14.707 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 4.8 M 2024-05-08T19:56:14.707 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 18 M 2024-05-08T19:56:14.707 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-05-08T19:56:14.707 INFO:teuthology.orchestra.run.smithi188.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-05-08T19:56:14.707 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-05-08T19:56:14.707 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-05-08T19:56:14.707 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-05-08T19:56:14.708 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-05-08T19:56:14.708 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-05-08T19:56:14.708 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-05-08T19:56:14.708 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-05-08T19:56:14.708 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-05-08T19:56:14.708 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-05-08T19:56:14.708 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-05-08T19:56:14.708 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-05-08T19:56:14.708 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-05-08T19:56:14.708 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-05-08T19:56:14.708 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-05-08T19:56:14.708 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-05-08T19:56:14.708 INFO:teuthology.orchestra.run.smithi188.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-05-08T19:56:14.709 INFO:teuthology.orchestra.run.smithi188.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-05-08T19:56:14.709 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-05-08T19:56:14.709 INFO:teuthology.orchestra.run.smithi188.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-05-08T19:56:14.709 INFO:teuthology.orchestra.run.smithi188.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-05-08T19:56:14.709 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-05-08T19:56:14.709 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-05-08T19:56:14.709 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-05-08T19:56:14.709 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-05-08T19:56:14.709 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:56:14.709 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-05-08T19:56:14.709 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:56:14.709 INFO:teuthology.orchestra.run.smithi188.stdout:Install 32 Packages 2024-05-08T19:56:14.709 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:56:14.710 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 31 M 2024-05-08T19:56:14.710 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 124 M 2024-05-08T19:56:14.710 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-05-08T19:56:15.266 INFO:teuthology.orchestra.run.smithi188.stdout:(1/32): ceph-18.2.2-1322.gd21e4f96.el8.x86_64.r 51 kB/s | 6.4 kB 00:00 2024-05-08T19:56:15.666 INFO:teuthology.orchestra.run.smithi188.stdout:(2/32): ceph-mgr-18.2.2-1322.gd21e4f96.el8.x86_ 3.0 MB/s | 1.6 MB 00:00 2024-05-08T19:56:15.833 INFO:teuthology.orchestra.run.smithi188.stdout:(3/32): ceph-mds-18.2.2-1322.gd21e4f96.el8.x86_ 3.3 MB/s | 2.3 MB 00:00 2024-05-08T19:56:15.909 INFO:teuthology.orchestra.run.smithi188.stdout:(4/32): ceph-mgr-modules-core-18.2.2-1322.gd21e 3.2 MB/s | 245 kB 00:00 2024-05-08T19:56:16.284 INFO:teuthology.orchestra.run.smithi188.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 419 kB/s | 157 kB 00:00 2024-05-08T19:56:16.376 INFO:teuthology.orchestra.run.smithi188.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 423 kB/s | 39 kB 00:00 2024-05-08T19:56:16.485 INFO:teuthology.orchestra.run.smithi188.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 955 kB/s | 103 kB 00:00 2024-05-08T19:56:16.585 INFO:teuthology.orchestra.run.smithi188.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 570 kB/s | 57 kB 00:00 2024-05-08T19:56:16.760 INFO:teuthology.orchestra.run.smithi188.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 2.6 MB/s | 457 kB 00:00 2024-05-08T19:56:16.902 INFO:teuthology.orchestra.run.smithi188.stdout:(10/32): ceph-mon-18.2.2-1322.gd21e4f96.el8.x86 3.0 MB/s | 4.8 MB 00:01 2024-05-08T19:56:17.094 INFO:teuthology.orchestra.run.smithi188.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 745 kB/s | 249 kB 00:00 2024-05-08T19:56:17.186 INFO:teuthology.orchestra.run.smithi188.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.2 MB/s | 109 kB 00:00 2024-05-08T19:56:17.670 INFO:teuthology.orchestra.run.smithi188.stdout:(13/32): ceph-osd-18.2.2-1322.gd21e4f96.el8.x86 9.2 MB/s | 18 MB 00:02 2024-05-08T19:56:17.695 INFO:teuthology.orchestra.run.smithi188.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 86 kB/s | 44 kB 00:00 2024-05-08T19:56:17.729 INFO:teuthology.orchestra.run.smithi188.stdout:(15/32): python3-cryptography-3.2.1-7.el8.x86_6 776 kB/s | 641 kB 00:00 2024-05-08T19:56:17.937 INFO:teuthology.orchestra.run.smithi188.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 694 kB/s | 185 kB 00:00 2024-05-08T19:56:18.046 INFO:teuthology.orchestra.run.smithi188.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 102 kB/s | 11 kB 00:00 2024-05-08T19:56:18.155 INFO:teuthology.orchestra.run.smithi188.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 162 kB/s | 18 kB 00:00 2024-05-08T19:56:18.180 INFO:teuthology.orchestra.run.smithi188.stdout:(19/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 356 kB/s | 173 kB 00:00 2024-05-08T19:56:18.240 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1322.gd21e4f96.el8.x86_64 5/6 2024-05-08T19:56:18.272 INFO:teuthology.orchestra.run.smithi188.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 416 kB/s | 49 kB 00:00 2024-05-08T19:56:18.297 INFO:teuthology.orchestra.run.smithi188.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 502 kB/s | 59 kB 00:00 2024-05-08T19:56:18.322 INFO:teuthology.orchestra.run.smithi188.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 648 kB/s | 384 kB 00:00 2024-05-08T19:56:18.398 INFO:teuthology.orchestra.run.smithi188.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 155 kB/s | 16 kB 00:00 2024-05-08T19:56:18.448 INFO:teuthology.orchestra.run.smithi188.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 149 kB/s | 19 kB 00:00 2024-05-08T19:56:18.473 INFO:teuthology.orchestra.run.smithi188.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2024-05-08T19:56:18.490 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-test-2:18.2.2-1322.gd21e4f96.el8.x86_64 6/6 2024-05-08T19:56:18.499 INFO:teuthology.orchestra.run.smithi188.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 238 kB/s | 24 kB 00:00 2024-05-08T19:56:18.541 INFO:teuthology.orchestra.run.smithi188.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 317 kB/s | 29 kB 00:00 2024-05-08T19:56:18.566 INFO:teuthology.orchestra.run.smithi188.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 298 kB/s | 27 kB 00:00 2024-05-08T19:56:18.692 INFO:teuthology.orchestra.run.smithi188.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.3 MB/s | 255 kB 00:00 2024-05-08T19:56:18.717 INFO:teuthology.orchestra.run.smithi188.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.4 MB/s | 251 kB 00:00 2024-05-08T19:56:18.742 INFO:teuthology.orchestra.run.smithi188.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 485 kB/s | 85 kB 00:00 2024-05-08T19:56:18.770 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-test-2:18.2.2-1322.gd21e4f96.el8.x86_64 6/6 2024-05-08T19:56:18.770 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-test-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/6 2024-05-08T19:56:18.771 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephsqlite-2:18.2.2-1322.gd21e4f96.el8.x86_64 2/6 2024-05-08T19:56:18.771 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-05-08T19:56:18.771 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-05-08T19:56:18.771 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-05-08T19:56:18.792 INFO:teuthology.orchestra.run.smithi188.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 229 kB/s | 23 kB 00:00 2024-05-08T19:56:18.794 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:56:18.794 INFO:teuthology.orchestra.run.smithi188.stdout:Total 7.6 MB/s | 31 MB 00:04 2024-05-08T19:56:18.876 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-05-08T19:56:18.913 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-05-08T19:56:18.913 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-05-08T19:56:19.213 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-05-08T19:56:19.213 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:56:19.213 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-05-08T19:56:19.213 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:56:19.213 INFO:teuthology.orchestra.run.smithi083.stdout: jq-1.6-8.el8.x86_64 2024-05-08T19:56:19.213 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:56:19.213 INFO:teuthology.orchestra.run.smithi083.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-05-08T19:56:19.213 INFO:teuthology.orchestra.run.smithi083.stdout: socat-1.7.4.1-1.el8.x86_64 2024-05-08T19:56:19.214 INFO:teuthology.orchestra.run.smithi083.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-05-08T19:56:19.214 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:56:19.214 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:56:19.244 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-05-08T19:56:19.247 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-05-08T19:56:19.420 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph 2024-05-08T19:56:20.160 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:09 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:56:20.356 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-05-08T19:56:20.579 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-05-08T19:56:20.790 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-05-08T19:56:21.032 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-05-08T19:56:21.222 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-05-08T19:56:21.445 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-05-08T19:56:21.624 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-05-08T19:56:21.818 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:56:21.822 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:56:21.822 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-05-08T19:56:21.822 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:56:21.822 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-05-08T19:56:21.822 INFO:teuthology.orchestra.run.smithi083.stdout: ceph x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 6.4 k 2024-05-08T19:56:21.822 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-05-08T19:56:21.822 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 2.3 M 2024-05-08T19:56:21.823 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 1.6 M 2024-05-08T19:56:21.823 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 245 k 2024-05-08T19:56:21.823 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 4.8 M 2024-05-08T19:56:21.823 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 18 M 2024-05-08T19:56:21.823 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-05-08T19:56:21.823 INFO:teuthology.orchestra.run.smithi083.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-05-08T19:56:21.823 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-05-08T19:56:21.823 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-05-08T19:56:21.823 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-05-08T19:56:21.823 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-05-08T19:56:21.823 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-05-08T19:56:21.823 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-05-08T19:56:21.824 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-05-08T19:56:21.824 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-05-08T19:56:21.824 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-05-08T19:56:21.824 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-05-08T19:56:21.824 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-05-08T19:56:21.824 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-05-08T19:56:21.824 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-05-08T19:56:21.824 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-05-08T19:56:21.824 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-05-08T19:56:21.824 INFO:teuthology.orchestra.run.smithi083.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-05-08T19:56:21.824 INFO:teuthology.orchestra.run.smithi083.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-05-08T19:56:21.824 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-05-08T19:56:21.824 INFO:teuthology.orchestra.run.smithi083.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-05-08T19:56:21.825 INFO:teuthology.orchestra.run.smithi083.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-05-08T19:56:21.825 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-05-08T19:56:21.825 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-05-08T19:56:21.825 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-05-08T19:56:21.825 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-05-08T19:56:21.825 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:56:21.825 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-05-08T19:56:21.825 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:56:21.825 INFO:teuthology.orchestra.run.smithi083.stdout:Install 32 Packages 2024-05-08T19:56:21.825 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:56:21.826 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 31 M 2024-05-08T19:56:21.826 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 124 M 2024-05-08T19:56:21.826 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-05-08T19:56:21.827 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-05-08T19:56:21.987 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-05-08T19:56:22.194 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-05-08T19:56:22.324 INFO:teuthology.orchestra.run.smithi083.stdout:(1/32): ceph-18.2.2-1322.gd21e4f96.el8.x86_64.r 27 kB/s | 6.4 kB 00:00 2024-05-08T19:56:22.474 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-05-08T19:56:22.702 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-05-08T19:56:22.926 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-05-08T19:56:23.199 INFO:teuthology.orchestra.run.smithi083.stdout:(2/32): ceph-mds-18.2.2-1322.gd21e4f96.el8.x86_ 2.0 MB/s | 2.3 MB 00:01 2024-05-08T19:56:23.305 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-05-08T19:56:23.568 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-05-08T19:56:23.591 INFO:teuthology.orchestra.run.smithi083.stdout:(3/32): ceph-mgr-18.2.2-1322.gd21e4f96.el8.x86_ 1.0 MB/s | 1.6 MB 00:01 2024-05-08T19:56:23.781 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-05-08T19:56:23.783 INFO:teuthology.orchestra.run.smithi083.stdout:(4/32): ceph-mgr-modules-core-18.2.2-1322.gd21e 1.3 MB/s | 245 kB 00:00 2024-05-08T19:56:23.997 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-05-08T19:56:24.158 INFO:teuthology.orchestra.run.smithi083.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 419 kB/s | 157 kB 00:00 2024-05-08T19:56:24.224 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-05-08T19:56:24.258 INFO:teuthology.orchestra.run.smithi083.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 388 kB/s | 39 kB 00:00 2024-05-08T19:56:24.351 INFO:teuthology.orchestra.run.smithi083.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.1 MB/s | 103 kB 00:00 2024-05-08T19:56:24.442 INFO:teuthology.orchestra.run.smithi083.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 623 kB/s | 57 kB 00:00 2024-05-08T19:56:24.525 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-05-08T19:56:24.611 INFO:teuthology.orchestra.run.smithi083.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 2.7 MB/s | 457 kB 00:00 2024-05-08T19:56:24.745 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-05-08T19:56:24.868 INFO:teuthology.orchestra.run.smithi083.stdout:(10/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 963 kB/s | 249 kB 00:00 2024-05-08T19:56:24.934 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-05-08T19:56:24.968 INFO:teuthology.orchestra.run.smithi083.stdout:(11/32): python3-cryptography-3.2.1-7.el8.x86_6 6.3 MB/s | 641 kB 00:00 2024-05-08T19:56:25.027 INFO:teuthology.orchestra.run.smithi083.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.8 MB/s | 109 kB 00:00 2024-05-08T19:56:25.144 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-05-08T19:56:25.407 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-05-08T19:56:25.536 INFO:teuthology.orchestra.run.smithi083.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 86 kB/s | 44 kB 00:00 2024-05-08T19:56:25.659 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-05-08T19:56:25.794 INFO:teuthology.orchestra.run.smithi083.stdout:(14/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 718 kB/s | 185 kB 00:00 2024-05-08T19:56:25.891 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-05-08T19:56:26.103 INFO:teuthology.orchestra.run.smithi083.stdout:(15/32): ceph-osd-18.2.2-1322.gd21e4f96.el8.x86 6.3 MB/s | 18 MB 00:02 2024-05-08T19:56:26.128 INFO:teuthology.orchestra.run.smithi083.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 517 kB/s | 173 kB 00:00 2024-05-08T19:56:26.237 INFO:teuthology.orchestra.run.smithi083.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 102 kB/s | 11 kB 00:00 2024-05-08T19:56:26.249 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-05-08T19:56:26.337 INFO:teuthology.orchestra.run.smithi083.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 176 kB/s | 18 kB 00:00 2024-05-08T19:56:26.446 INFO:teuthology.orchestra.run.smithi083.stdout:(19/32): python3-logutils-0.3.5-11.el8.noarch.r 450 kB/s | 49 kB 00:00 2024-05-08T19:56:26.638 INFO:teuthology.orchestra.run.smithi083.stdout:(20/32): python3-more-itertools-7.2.0-3.el8.noa 307 kB/s | 59 kB 00:00 2024-05-08T19:56:26.813 INFO:teuthology.orchestra.run.smithi083.stdout:(21/32): ceph-mon-18.2.2-1322.gd21e4f96.el8.x86 1.1 MB/s | 4.8 MB 00:04 2024-05-08T19:56:27.001 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-05-08T19:56:27.297 INFO:teuthology.orchestra.run.smithi083.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 322 kB/s | 384 kB 00:01 2024-05-08T19:56:27.322 INFO:teuthology.orchestra.run.smithi083.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 413 kB/s | 283 kB 00:00 2024-05-08T19:56:27.386 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1322.gd21e4f96.el8. 27/32 2024-05-08T19:56:27.406 INFO:teuthology.orchestra.run.smithi083.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 173 kB/s | 19 kB 00:00 2024-05-08T19:56:27.431 INFO:teuthology.orchestra.run.smithi083.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 220 kB/s | 24 kB 00:00 2024-05-08T19:56:27.482 INFO:teuthology.orchestra.run.smithi083.stdout:(26/32): python3-portend-2.6-1.el8.noarch.rpm 23 kB/s | 16 kB 00:00 2024-05-08T19:56:27.515 INFO:teuthology.orchestra.run.smithi083.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 269 kB/s | 29 kB 00:00 2024-05-08T19:56:27.540 INFO:teuthology.orchestra.run.smithi083.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 253 kB/s | 27 kB 00:00 2024-05-08T19:56:27.598 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-2:18.2.2-1322.gd21e4f96.el8.x86_64 28/32 2024-05-08T19:56:27.732 INFO:teuthology.orchestra.run.smithi083.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 445 kB/s | 85 kB 00:00 2024-05-08T19:56:27.841 INFO:teuthology.orchestra.run.smithi083.stdout:(30/32): python3-zc-lockfile-2.0-2.el8.noarch.r 212 kB/s | 23 kB 00:00 2024-05-08T19:56:27.874 INFO:teuthology.orchestra.run.smithi083.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 699 kB/s | 251 kB 00:00 2024-05-08T19:56:27.916 INFO:teuthology.orchestra.run.smithi083.stdout:(32/32): python3-waitress-1.4.3-1.el8.noarch.rp 588 kB/s | 255 kB 00:00 2024-05-08T19:56:27.921 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:56:27.921 INFO:teuthology.orchestra.run.smithi083.stdout:Total 5.1 MB/s | 31 MB 00:06 2024-05-08T19:56:28.000 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-05-08T19:56:28.037 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-05-08T19:56:28.038 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-05-08T19:56:28.358 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-05-08T19:56:28.361 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-05-08T19:56:29.249 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1322.gd21e4f96.el8.x86_64 28/32 2024-05-08T19:56:29.431 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-osd-2:18.2.2-1322.gd21e4f96.el8.x86_64 29/32 2024-05-08T19:56:29.455 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-05-08T19:56:29.670 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-05-08T19:56:29.872 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-05-08T19:56:29.908 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-osd-2:18.2.2-1322.gd21e4f96.el8.x86_64 29/32 2024-05-08T19:56:30.106 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mon-2:18.2.2-1322.gd21e4f96.el8.x86_64 30/32 2024-05-08T19:56:30.114 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-05-08T19:56:30.313 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-05-08T19:56:30.318 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mon-2:18.2.2-1322.gd21e4f96.el8.x86_64 30/32 2024-05-08T19:56:30.480 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mds-2:18.2.2-1322.gd21e4f96.el8.x86_64 31/32 2024-05-08T19:56:30.498 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mds-2:18.2.2-1322.gd21e4f96.el8.x86_64 31/32 2024-05-08T19:56:30.519 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-05-08T19:56:30.605 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-2:18.2.2-1322.gd21e4f96.el8.x86_64 32/32 2024-05-08T19:56:30.715 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-05-08T19:56:30.892 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-05-08T19:56:31.004 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-2:18.2.2-1322.gd21e4f96.el8.x86_64 32/32 2024-05-08T19:56:31.004 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/32 2024-05-08T19:56:31.004 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mds-2:18.2.2-1322.gd21e4f96.el8.x86_64 2/32 2024-05-08T19:56:31.004 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-2:18.2.2-1322.gd21e4f96.el8.x86_64 3/32 2024-05-08T19:56:31.004 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mon-2:18.2.2-1322.gd21e4f96.el8.x86_64 4/32 2024-05-08T19:56:31.004 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-osd-2:18.2.2-1322.gd21e4f96.el8.x86_64 5/32 2024-05-08T19:56:31.004 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1322.gd21e4f96.el8. 6/32 2024-05-08T19:56:31.004 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-05-08T19:56:31.004 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-05-08T19:56:31.005 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-05-08T19:56:31.005 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-05-08T19:56:31.005 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-05-08T19:56:31.005 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-05-08T19:56:31.005 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-05-08T19:56:31.005 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-05-08T19:56:31.005 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-05-08T19:56:31.005 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-05-08T19:56:31.005 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-05-08T19:56:31.005 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-05-08T19:56:31.005 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-05-08T19:56:31.005 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-05-08T19:56:31.005 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-05-08T19:56:31.005 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-05-08T19:56:31.006 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-05-08T19:56:31.006 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-05-08T19:56:31.006 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-05-08T19:56:31.006 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-05-08T19:56:31.006 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-05-08T19:56:31.006 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-05-08T19:56:31.006 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-05-08T19:56:31.006 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-05-08T19:56:31.006 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-05-08T19:56:31.070 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-05-08T19:56:31.328 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-05-08T19:56:31.565 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-05-08T19:56:31.812 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-05-08T19:56:31.971 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-05-08T19:56:31.972 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:56:31.972 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-05-08T19:56:31.972 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:56:31.972 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:56:31.972 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:56:31.972 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T19:56:31.972 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:56:31.972 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:56:31.972 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-05-08T19:56:31.972 INFO:teuthology.orchestra.run.smithi188.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-05-08T19:56:31.973 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-05-08T19:56:31.973 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-05-08T19:56:31.973 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-05-08T19:56:31.973 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-05-08T19:56:31.973 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-6.2-6.el8.noarch 2024-05-08T19:56:31.973 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-05-08T19:56:31.973 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-05-08T19:56:31.973 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako-1.0.6-14.el8.noarch 2024-05-08T19:56:31.973 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-05-08T19:56:31.973 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-05-08T19:56:31.973 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-05-08T19:56:31.973 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend-2.6-1.el8.noarch 2024-05-08T19:56:31.973 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-05-08T19:56:31.973 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser-2.14-14.el8.noarch 2024-05-08T19:56:31.974 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz-2017.2-11.el8.noarch 2024-05-08T19:56:31.974 INFO:teuthology.orchestra.run.smithi188.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-05-08T19:56:31.974 INFO:teuthology.orchestra.run.smithi188.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-05-08T19:56:31.974 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-05-08T19:56:31.974 INFO:teuthology.orchestra.run.smithi188.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-05-08T19:56:31.974 INFO:teuthology.orchestra.run.smithi188.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-05-08T19:56:31.974 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-05-08T19:56:31.974 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-05-08T19:56:31.974 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-05-08T19:56:31.974 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-05-08T19:56:31.974 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:56:31.974 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:56:32.044 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-05-08T19:56:32.232 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-base 2024-05-08T19:56:32.406 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-05-08T19:56:32.642 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-05-08T19:56:32.814 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-05-08T19:56:32.984 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:32 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:56:33.062 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-05-08T19:56:33.290 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-05-08T19:56:33.622 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-05-08T19:56:33.820 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-05-08T19:56:34.041 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-05-08T19:56:34.211 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-05-08T19:56:34.465 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-05-08T19:56:34.588 INFO:teuthology.orchestra.run.smithi188.stdout:Package ceph-base-2:18.2.2-1322.gd21e4f96.el8.x86_64 is already installed. 2024-05-08T19:56:34.636 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:56:34.637 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-05-08T19:56:34.637 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:56:34.704 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-05-08T19:56:34.731 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install cephadm 2024-05-08T19:56:34.933 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-05-08T19:56:35.266 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-05-08T19:56:35.482 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:35 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:56:35.586 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-05-08T19:56:35.987 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1322.gd21e4f96.el8. 27/32 2024-05-08T19:56:36.183 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-2:18.2.2-1322.gd21e4f96.el8.x86_64 28/32 2024-05-08T19:56:37.130 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:56:37.131 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:56:37.131 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-05-08T19:56:37.131 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:56:37.131 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-05-08T19:56:37.131 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 170 k 2024-05-08T19:56:37.131 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:56:37.131 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-05-08T19:56:37.131 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:56:37.132 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-05-08T19:56:37.132 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:56:37.132 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 170 k 2024-05-08T19:56:37.132 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 692 k 2024-05-08T19:56:37.132 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-05-08T19:56:37.309 INFO:teuthology.orchestra.run.smithi188.stdout:cephadm-18.2.2-1322.gd21e4f96.el8.noarch.rpm 965 kB/s | 170 kB 00:00 2024-05-08T19:56:37.309 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:56:37.309 INFO:teuthology.orchestra.run.smithi188.stdout:Total 958 kB/s | 170 kB 00:00 2024-05-08T19:56:37.310 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-05-08T19:56:37.317 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-05-08T19:56:37.318 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-05-08T19:56:37.320 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-05-08T19:56:37.323 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-05-08T19:56:37.365 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-05-08T19:56:37.832 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1322.gd21e4f96.el8.x86_64 28/32 2024-05-08T19:56:37.902 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 1/1 2024-05-08T19:56:38.024 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-osd-2:18.2.2-1322.gd21e4f96.el8.x86_64 29/32 2024-05-08T19:56:38.075 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 1/1 2024-05-08T19:56:38.299 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 1/1 2024-05-08T19:56:38.501 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-osd-2:18.2.2-1322.gd21e4f96.el8.x86_64 29/32 2024-05-08T19:56:38.661 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 1/1 2024-05-08T19:56:38.661 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:56:38.661 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-05-08T19:56:38.661 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T19:56:38.661 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:56:38.662 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:56:38.680 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mon-2:18.2.2-1322.gd21e4f96.el8.x86_64 30/32 2024-05-08T19:56:38.854 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-immutable-object-cache 2024-05-08T19:56:38.892 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mon-2:18.2.2-1322.gd21e4f96.el8.x86_64 30/32 2024-05-08T19:56:39.039 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mds-2:18.2.2-1322.gd21e4f96.el8.x86_64 31/32 2024-05-08T19:56:39.056 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mds-2:18.2.2-1322.gd21e4f96.el8.x86_64 31/32 2024-05-08T19:56:39.154 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-2:18.2.2-1322.gd21e4f96.el8.x86_64 32/32 2024-05-08T19:56:39.593 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-2:18.2.2-1322.gd21e4f96.el8.x86_64 32/32 2024-05-08T19:56:39.593 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/32 2024-05-08T19:56:39.593 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mds-2:18.2.2-1322.gd21e4f96.el8.x86_64 2/32 2024-05-08T19:56:39.593 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-2:18.2.2-1322.gd21e4f96.el8.x86_64 3/32 2024-05-08T19:56:39.593 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mon-2:18.2.2-1322.gd21e4f96.el8.x86_64 4/32 2024-05-08T19:56:39.593 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-osd-2:18.2.2-1322.gd21e4f96.el8.x86_64 5/32 2024-05-08T19:56:39.593 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1322.gd21e4f96.el8. 6/32 2024-05-08T19:56:39.594 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-05-08T19:56:39.594 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-05-08T19:56:39.594 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-05-08T19:56:39.594 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-05-08T19:56:39.594 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-05-08T19:56:39.594 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-05-08T19:56:39.594 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-05-08T19:56:39.594 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-05-08T19:56:39.594 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-05-08T19:56:39.594 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-05-08T19:56:39.594 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-05-08T19:56:39.594 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-05-08T19:56:39.594 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-05-08T19:56:39.595 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-05-08T19:56:39.595 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-05-08T19:56:39.595 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-05-08T19:56:39.595 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-05-08T19:56:39.595 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-05-08T19:56:39.595 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-05-08T19:56:39.595 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-05-08T19:56:39.595 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-05-08T19:56:39.595 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-05-08T19:56:39.595 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-05-08T19:56:39.595 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-05-08T19:56:39.595 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-05-08T19:56:39.616 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:39 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:56:40.521 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-05-08T19:56:40.521 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:56:40.521 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-05-08T19:56:40.521 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:56:40.521 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:56:40.521 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:56:40.521 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T19:56:40.522 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:56:40.522 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:56:40.522 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-05-08T19:56:40.522 INFO:teuthology.orchestra.run.smithi083.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-05-08T19:56:40.522 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-05-08T19:56:40.522 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-05-08T19:56:40.522 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-05-08T19:56:40.522 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-05-08T19:56:40.522 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-6.2-6.el8.noarch 2024-05-08T19:56:40.522 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-05-08T19:56:40.522 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-05-08T19:56:40.522 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako-1.0.6-14.el8.noarch 2024-05-08T19:56:40.522 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-05-08T19:56:40.523 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-05-08T19:56:40.523 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-05-08T19:56:40.523 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend-2.6-1.el8.noarch 2024-05-08T19:56:40.523 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-05-08T19:56:40.523 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser-2.14-14.el8.noarch 2024-05-08T19:56:40.523 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz-2017.2-11.el8.noarch 2024-05-08T19:56:40.523 INFO:teuthology.orchestra.run.smithi083.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-05-08T19:56:40.523 INFO:teuthology.orchestra.run.smithi083.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-05-08T19:56:40.523 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-05-08T19:56:40.523 INFO:teuthology.orchestra.run.smithi083.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-05-08T19:56:40.523 INFO:teuthology.orchestra.run.smithi083.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-05-08T19:56:40.523 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-05-08T19:56:40.523 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-05-08T19:56:40.524 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-05-08T19:56:40.524 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-05-08T19:56:40.524 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:56:40.524 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:56:40.763 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-base 2024-05-08T19:56:41.275 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:56:41.276 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:56:41.276 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-05-08T19:56:41.276 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:56:41.276 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-05-08T19:56:41.276 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 166 k 2024-05-08T19:56:41.276 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:56:41.276 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-05-08T19:56:41.276 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:56:41.276 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-05-08T19:56:41.276 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:56:41.277 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 166 k 2024-05-08T19:56:41.277 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 503 k 2024-05-08T19:56:41.277 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-05-08T19:56:41.490 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-immutable-object-cache-18.2.2-1322.gd21e4f 781 kB/s | 166 kB 00:00 2024-05-08T19:56:41.490 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:56:41.490 INFO:teuthology.orchestra.run.smithi188.stdout:Total 776 kB/s | 166 kB 00:00 2024-05-08T19:56:41.491 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-05-08T19:56:41.502 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-05-08T19:56:41.502 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-05-08T19:56:41.517 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:30 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:56:41.549 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-05-08T19:56:41.552 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-05-08T19:56:41.671 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-05-08T19:56:41.878 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1322.gd21e4f96. 1/1 2024-05-08T19:56:42.285 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1322.gd21e4f96. 1/1 2024-05-08T19:56:43.112 INFO:teuthology.orchestra.run.smithi083.stdout:Package ceph-base-2:18.2.2-1322.gd21e4f96.el8.x86_64 is already installed. 2024-05-08T19:56:43.160 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:56:43.161 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-05-08T19:56:43.161 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:56:43.261 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install cephadm 2024-05-08T19:56:43.412 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1322.gd21e4f96. 1/1 2024-05-08T19:56:43.412 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:56:43.412 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-05-08T19:56:43.412 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:56:43.412 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:56:43.413 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:56:44.006 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:33 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:56:44.517 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr 2024-05-08T19:56:45.287 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:45 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:56:45.652 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:56:45.653 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:56:45.653 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-05-08T19:56:45.653 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:56:45.653 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-05-08T19:56:45.653 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 170 k 2024-05-08T19:56:45.653 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:56:45.653 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-05-08T19:56:45.653 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:56:45.653 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-05-08T19:56:45.653 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:56:45.654 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 170 k 2024-05-08T19:56:45.654 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 692 k 2024-05-08T19:56:45.654 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-05-08T19:56:46.247 INFO:teuthology.orchestra.run.smithi083.stdout:cephadm-18.2.2-1322.gd21e4f96.el8.noarch.rpm 287 kB/s | 170 kB 00:00 2024-05-08T19:56:46.248 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:56:46.248 INFO:teuthology.orchestra.run.smithi083.stdout:Total 286 kB/s | 170 kB 00:00 2024-05-08T19:56:46.248 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-05-08T19:56:46.256 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-05-08T19:56:46.256 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-05-08T19:56:46.258 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-05-08T19:56:46.261 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-05-08T19:56:46.306 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-05-08T19:56:46.833 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 1/1 2024-05-08T19:56:46.884 INFO:teuthology.orchestra.run.smithi188.stdout:Package ceph-mgr-2:18.2.2-1322.gd21e4f96.el8.x86_64 is already installed. 2024-05-08T19:56:46.933 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:56:46.934 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-05-08T19:56:46.934 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:56:46.993 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 1/1 2024-05-08T19:56:47.069 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-dashboard 2024-05-08T19:56:47.218 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 1/1 2024-05-08T19:56:47.597 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 1/1 2024-05-08T19:56:47.597 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:56:47.597 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-05-08T19:56:47.597 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T19:56:47.597 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:56:47.597 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:56:47.816 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:47 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:56:47.838 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-immutable-object-cache 2024-05-08T19:56:48.613 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:37 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:56:49.438 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:56:49.439 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:56:49.439 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-05-08T19:56:49.439 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:56:49.439 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-05-08T19:56:49.439 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 3.5 M 2024-05-08T19:56:49.439 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-05-08T19:56:49.439 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 24 k 2024-05-08T19:56:49.439 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 16 k 2024-05-08T19:56:49.440 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-05-08T19:56:49.440 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-05-08T19:56:49.440 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:56:49.440 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-05-08T19:56:49.440 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:56:49.440 INFO:teuthology.orchestra.run.smithi188.stdout:Install 5 Packages 2024-05-08T19:56:49.440 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:56:49.441 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 3.8 M 2024-05-08T19:56:49.441 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 87 M 2024-05-08T19:56:49.441 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-05-08T19:56:50.027 INFO:teuthology.orchestra.run.smithi188.stdout:(1/5): ceph-grafana-dashboards-18.2.2-1322.gd21 55 kB/s | 24 kB 00:00 2024-05-08T19:56:50.052 INFO:teuthology.orchestra.run.smithi188.stdout:(2/5): ceph-prometheus-alerts-18.2.2-1322.gd21e 35 kB/s | 16 kB 00:00 2024-05-08T19:56:50.178 INFO:teuthology.orchestra.run.smithi188.stdout:(3/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 224 kB/s | 34 kB 00:00 2024-05-08T19:56:50.226 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:56:50.227 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:56:50.227 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-05-08T19:56:50.227 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:56:50.227 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-05-08T19:56:50.228 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 166 k 2024-05-08T19:56:50.228 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:56:50.228 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-05-08T19:56:50.228 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:56:50.228 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-05-08T19:56:50.228 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:56:50.228 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 166 k 2024-05-08T19:56:50.228 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 503 k 2024-05-08T19:56:50.229 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-05-08T19:56:50.328 INFO:teuthology.orchestra.run.smithi188.stdout:(4/5): ceph-mgr-dashboard-18.2.2-1322.gd21e4f96 4.7 MB/s | 3.5 MB 00:00 2024-05-08T19:56:50.362 INFO:teuthology.orchestra.run.smithi188.stdout:(5/5): python3-routes-2.4.1-12.el8.noarch.rpm 636 kB/s | 196 kB 00:00 2024-05-08T19:56:50.363 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:56:50.364 INFO:teuthology.orchestra.run.smithi188.stdout:Total 4.1 MB/s | 3.8 MB 00:00 2024-05-08T19:56:50.364 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-05-08T19:56:50.372 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-05-08T19:56:50.372 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-05-08T19:56:50.412 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-immutable-object-cache-18.2.2-1322.gd21e4f 906 kB/s | 166 kB 00:00 2024-05-08T19:56:50.412 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:56:50.413 INFO:teuthology.orchestra.run.smithi083.stdout:Total 899 kB/s | 166 kB 00:00 2024-05-08T19:56:50.413 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-05-08T19:56:50.424 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-05-08T19:56:50.424 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-05-08T19:56:50.450 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-05-08T19:56:50.453 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-05-08T19:56:50.471 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-05-08T19:56:50.474 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-05-08T19:56:50.578 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-05-08T19:56:50.688 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-05-08T19:56:50.754 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1322.gd21e4f96. 1/1 2024-05-08T19:56:50.907 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-05-08T19:56:51.103 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-05-08T19:56:51.145 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1322.gd21e4f96. 1/1 2024-05-08T19:56:51.256 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1322.gd21e4f96.el8.n 3/5 2024-05-08T19:56:51.507 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1322.gd21e4f96. 1/1 2024-05-08T19:56:51.507 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:56:51.507 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-05-08T19:56:51.508 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:56:51.508 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:56:51.508 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:56:51.721 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr 2024-05-08T19:56:52.386 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1322.gd21e4f96.el8. 4/5 2024-05-08T19:56:52.482 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:41 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:56:52.608 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1322.gd21e4f96.el8.noarc 5/5 2024-05-08T19:56:52.886 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1322.gd21e4f96.el8.noarc 5/5 2024-05-08T19:56:52.886 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1322.gd21e4f96.el8. 1/5 2024-05-08T19:56:52.886 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1322.gd21e4f96.el8.noarc 2/5 2024-05-08T19:56:52.886 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1322.gd21e4f96.el8.n 3/5 2024-05-08T19:56:52.886 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-05-08T19:56:53.271 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-05-08T19:56:53.271 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:56:53.271 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-05-08T19:56:53.271 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T19:56:53.271 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T19:56:53.271 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T19:56:53.272 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-05-08T19:56:53.272 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes-2.4.1-12.el8.noarch 2024-05-08T19:56:53.272 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:56:53.272 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:56:53.382 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-08T19:56:54.087 INFO:teuthology.orchestra.run.smithi083.stdout:Package ceph-mgr-2:18.2.2-1322.gd21e4f96.el8.x86_64 is already installed. 2024-05-08T19:56:54.135 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:54 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:56:54.135 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:56:54.137 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-05-08T19:56:54.137 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:56:54.242 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-dashboard 2024-05-08T19:56:54.983 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:43 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:56:55.759 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:56:55.760 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================ 2024-05-08T19:56:55.760 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-05-08T19:56:55.760 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================ 2024-05-08T19:56:55.760 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-05-08T19:56:55.760 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 7.4 M 2024-05-08T19:56:55.760 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-05-08T19:56:55.761 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran x86_64 8.5.0-22.el8 baseos 645 k 2024-05-08T19:56:55.761 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath x86_64 8.5.0-22.el8 baseos 172 k 2024-05-08T19:56:55.761 INFO:teuthology.orchestra.run.smithi188.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-05-08T19:56:55.761 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-05-08T19:56:55.761 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-05-08T19:56:55.761 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-05-08T19:56:55.761 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-05-08T19:56:55.761 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:56:55.761 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-05-08T19:56:55.761 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================ 2024-05-08T19:56:55.761 INFO:teuthology.orchestra.run.smithi188.stdout:Install 8 Packages 2024-05-08T19:56:55.761 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:56:55.762 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 36 M 2024-05-08T19:56:55.762 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 210 M 2024-05-08T19:56:55.762 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-05-08T19:56:56.594 INFO:teuthology.orchestra.run.smithi188.stdout:(1/8): ceph-mgr-diskprediction-local-18.2.2-132 13 MB/s | 7.4 MB 00:00 2024-05-08T19:56:56.646 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:56:56.646 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:56:56.647 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-05-08T19:56:56.647 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:56:56.647 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-05-08T19:56:56.647 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 3.5 M 2024-05-08T19:56:56.647 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-05-08T19:56:56.647 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 24 k 2024-05-08T19:56:56.647 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 16 k 2024-05-08T19:56:56.647 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-05-08T19:56:56.647 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-05-08T19:56:56.647 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:56:56.647 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-05-08T19:56:56.647 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:56:56.647 INFO:teuthology.orchestra.run.smithi083.stdout:Install 5 Packages 2024-05-08T19:56:56.647 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:56:56.648 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 3.8 M 2024-05-08T19:56:56.648 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 87 M 2024-05-08T19:56:56.648 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-05-08T19:56:56.861 INFO:teuthology.orchestra.run.smithi188.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.7 MB/s | 4.9 MB 00:00 2024-05-08T19:56:56.953 INFO:teuthology.orchestra.run.smithi188.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 5.0 MB/s | 4.7 MB 00:00 2024-05-08T19:56:56.965 INFO:teuthology.orchestra.run.smithi083.stdout:(1/5): ceph-grafana-dashboards-18.2.2-1322.gd21 169 kB/s | 24 kB 00:00 2024-05-08T19:56:56.978 INFO:teuthology.orchestra.run.smithi188.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.9 MB/s | 225 kB 00:00 2024-05-08T19:56:57.066 INFO:teuthology.orchestra.run.smithi083.stdout:(2/5): ceph-prometheus-alerts-18.2.2-1322.gd21e 67 kB/s | 16 kB 00:00 2024-05-08T19:56:57.125 INFO:teuthology.orchestra.run.smithi083.stdout:(3/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 212 kB/s | 34 kB 00:00 2024-05-08T19:56:57.228 INFO:teuthology.orchestra.run.smithi188.stdout:(5/8): libgfortran-8.5.0-22.el8.x86_64.rpm 2.5 MB/s | 645 kB 00:00 2024-05-08T19:56:57.283 INFO:teuthology.orchestra.run.smithi083.stdout:(4/5): python3-routes-2.4.1-12.el8.noarch.rpm 906 kB/s | 196 kB 00:00 2024-05-08T19:56:57.287 INFO:teuthology.orchestra.run.smithi188.stdout:(6/8): libquadmath-8.5.0-22.el8.x86_64.rpm 2.9 MB/s | 172 kB 00:00 2024-05-08T19:56:57.387 INFO:teuthology.orchestra.run.smithi188.stdout:(7/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.7 MB/s | 3.7 MB 00:00 2024-05-08T19:56:57.500 INFO:teuthology.orchestra.run.smithi083.stdout:(5/5): ceph-mgr-dashboard-18.2.2-1322.gd21e4f96 5.2 MB/s | 3.5 MB 00:00 2024-05-08T19:56:57.501 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:56:57.502 INFO:teuthology.orchestra.run.smithi083.stdout:Total 4.4 MB/s | 3.8 MB 00:00 2024-05-08T19:56:57.502 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-05-08T19:56:57.510 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-05-08T19:56:57.510 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-05-08T19:56:57.591 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-05-08T19:56:57.594 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-05-08T19:56:57.830 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-05-08T19:56:57.891 INFO:teuthology.orchestra.run.smithi188.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 15 MB/s | 14 MB 00:00 2024-05-08T19:56:57.891 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:56:57.891 INFO:teuthology.orchestra.run.smithi188.stdout:Total 17 MB/s | 36 MB 00:02 2024-05-08T19:56:58.016 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-05-08T19:56:58.122 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-05-08T19:56:58.140 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-05-08T19:56:58.141 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-05-08T19:56:58.205 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-05-08T19:56:58.334 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1322.gd21e4f96.el8.n 3/5 2024-05-08T19:56:58.526 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-05-08T19:56:58.529 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-05-08T19:56:59.076 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-05-08T19:56:59.223 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libquadmath-8.5.0-22.el8.x86_64 1/8 2024-05-08T19:56:59.379 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libquadmath-8.5.0-22.el8.x86_64 1/8 2024-05-08T19:56:59.487 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1322.gd21e4f96.el8. 4/5 2024-05-08T19:56:59.532 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libgfortran-8.5.0-22.el8.x86_64 2/8 2024-05-08T19:56:59.729 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1322.gd21e4f96.el8.noarc 5/5 2024-05-08T19:57:00.015 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1322.gd21e4f96.el8.noarc 5/5 2024-05-08T19:57:00.016 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1322.gd21e4f96.el8. 1/5 2024-05-08T19:57:00.016 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1322.gd21e4f96.el8.noarc 2/5 2024-05-08T19:57:00.016 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1322.gd21e4f96.el8.n 3/5 2024-05-08T19:57:00.016 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-05-08T19:57:00.114 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libgfortran-8.5.0-22.el8.x86_64 2/8 2024-05-08T19:57:00.397 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-05-08T19:57:00.397 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:00.397 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-05-08T19:57:00.397 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T19:57:00.397 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T19:57:00.397 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T19:57:00.397 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-05-08T19:57:00.397 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes-2.4.1-12.el8.noarch 2024-05-08T19:57:00.398 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:00.398 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:57:00.523 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-08T19:57:01.048 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-05-08T19:57:01.272 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:50 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:57:01.670 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-05-08T19:57:01.831 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-05-08T19:57:02.515 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-05-08T19:57:02.842 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-05-08T19:57:02.916 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:57:02.918 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================================ 2024-05-08T19:57:02.918 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-05-08T19:57:02.918 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================================ 2024-05-08T19:57:02.918 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-05-08T19:57:02.918 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 7.4 M 2024-05-08T19:57:02.918 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-05-08T19:57:02.918 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran x86_64 8.5.0-22.el8 baseos 645 k 2024-05-08T19:57:02.918 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath x86_64 8.5.0-22.el8 baseos 172 k 2024-05-08T19:57:02.918 INFO:teuthology.orchestra.run.smithi083.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-05-08T19:57:02.918 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-05-08T19:57:02.918 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-05-08T19:57:02.918 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-05-08T19:57:02.919 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-05-08T19:57:02.919 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:02.919 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-05-08T19:57:02.919 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================================ 2024-05-08T19:57:02.919 INFO:teuthology.orchestra.run.smithi083.stdout:Install 8 Packages 2024-05-08T19:57:02.919 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:02.919 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 36 M 2024-05-08T19:57:02.920 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 210 M 2024-05-08T19:57:02.920 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-05-08T19:57:03.089 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-05-08T19:57:04.173 INFO:teuthology.orchestra.run.smithi083.stdout:(1/8): ceph-mgr-diskprediction-local-18.2.2-132 13 MB/s | 7.4 MB 00:00 2024-05-08T19:57:04.450 INFO:teuthology.orchestra.run.smithi083.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.7 MB/s | 4.9 MB 00:00 2024-05-08T19:57:04.515 INFO:teuthology.orchestra.run.smithi083.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 5.2 MB/s | 4.7 MB 00:00 2024-05-08T19:57:04.540 INFO:teuthology.orchestra.run.smithi083.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2024-05-08T19:57:04.791 INFO:teuthology.orchestra.run.smithi083.stdout:(5/8): libgfortran-8.5.0-22.el8.x86_64.rpm 2.5 MB/s | 645 kB 00:00 2024-05-08T19:57:04.854 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-05-08T19:57:04.858 INFO:teuthology.orchestra.run.smithi083.stdout:(6/8): libquadmath-8.5.0-22.el8.x86_64.rpm 2.5 MB/s | 172 kB 00:00 2024-05-08T19:57:04.966 INFO:teuthology.orchestra.run.smithi083.stdout:(7/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.7 MB/s | 3.7 MB 00:00 2024-05-08T19:57:05.627 INFO:teuthology.orchestra.run.smithi083.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 13 MB/s | 14 MB 00:01 2024-05-08T19:57:05.627 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:57:05.628 INFO:teuthology.orchestra.run.smithi083.stdout:Total 13 MB/s | 36 MB 00:02 2024-05-08T19:57:05.860 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-05-08T19:57:05.878 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-05-08T19:57:05.878 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-05-08T19:57:06.112 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-05-08T19:57:06.261 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-05-08T19:57:06.264 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-05-08T19:57:06.313 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1322.gd21e4f9 8/8 2024-05-08T19:57:06.640 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1322.gd21e4f9 8/8 2024-05-08T19:57:06.640 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1322.gd21e4f9 1/8 2024-05-08T19:57:06.640 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-05-08T19:57:06.640 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-05-08T19:57:06.640 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-05-08T19:57:06.640 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-05-08T19:57:06.641 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-05-08T19:57:06.641 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libgfortran-8.5.0-22.el8.x86_64 7/8 2024-05-08T19:57:06.839 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-05-08T19:57:06.985 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libquadmath-8.5.0-22.el8.x86_64 1/8 2024-05-08T19:57:07.060 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libquadmath-8.5.0-22.el8.x86_64 8/8 2024-05-08T19:57:07.060 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:07.060 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-05-08T19:57:07.060 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T19:57:07.060 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran-8.5.0-22.el8.x86_64 2024-05-08T19:57:07.060 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath-8.5.0-22.el8.x86_64 2024-05-08T19:57:07.060 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-0.3.15-4.el8.x86_64 2024-05-08T19:57:07.060 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-05-08T19:57:07.060 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-05-08T19:57:07.060 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-05-08T19:57:07.061 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-05-08T19:57:07.061 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:07.061 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:57:07.128 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libquadmath-8.5.0-22.el8.x86_64 1/8 2024-05-08T19:57:07.184 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-rook 2024-05-08T19:57:07.276 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libgfortran-8.5.0-22.el8.x86_64 2/8 2024-05-08T19:57:07.857 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libgfortran-8.5.0-22.el8.x86_64 2/8 2024-05-08T19:57:07.939 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:07 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:57:08.051 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-05-08T19:57:08.659 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-05-08T19:57:08.818 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-05-08T19:57:09.498 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-05-08T19:57:09.587 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:57:09.589 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:57:09.589 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-05-08T19:57:09.589 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:57:09.589 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-05-08T19:57:09.589 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 49 k 2024-05-08T19:57:09.589 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-05-08T19:57:09.590 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-05-08T19:57:09.590 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-05-08T19:57:09.590 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-05-08T19:57:09.590 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-05-08T19:57:09.590 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-05-08T19:57:09.590 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-05-08T19:57:09.590 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-05-08T19:57:09.590 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-05-08T19:57:09.590 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-05-08T19:57:09.590 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-05-08T19:57:09.590 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-05-08T19:57:09.590 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-05-08T19:57:09.590 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-05-08T19:57:09.591 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:09.591 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-05-08T19:57:09.591 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:57:09.591 INFO:teuthology.orchestra.run.smithi188.stdout:Install 14 Packages 2024-05-08T19:57:09.591 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:09.591 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 2.0 M 2024-05-08T19:57:09.591 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 25 M 2024-05-08T19:57:09.591 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-05-08T19:57:09.787 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-05-08T19:57:10.020 INFO:teuthology.orchestra.run.smithi188.stdout:(1/14): ceph-mgr-rook-18.2.2-1322.gd21e4f96.el8 298 kB/s | 49 kB 00:00 2024-05-08T19:57:10.068 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-05-08T19:57:10.079 INFO:teuthology.orchestra.run.smithi188.stdout:(2/14): python3-jsonpointer-1.10-11.el8.noarch. 89 kB/s | 20 kB 00:00 2024-05-08T19:57:10.104 INFO:teuthology.orchestra.run.smithi188.stdout:(3/14): python3-jsonpatch-1.21-2.el8.noarch.rpm 107 kB/s | 27 kB 00:00 2024-05-08T19:57:10.263 INFO:teuthology.orchestra.run.smithi188.stdout:(4/14): python3-jwt-1.6.1-2.el8.noarch.rpm 269 kB/s | 43 kB 00:00 2024-05-08T19:57:10.305 INFO:teuthology.orchestra.run.smithi188.stdout:(5/14): python3-pyasn1-modules-0.3.7-6.el8.noar 487 kB/s | 110 kB 00:00 2024-05-08T19:57:10.372 INFO:teuthology.orchestra.run.smithi188.stdout:(6/14): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.4 MB/s | 155 kB 00:00 2024-05-08T19:57:10.397 INFO:teuthology.orchestra.run.smithi188.stdout:(7/14): python3-pyasn1-0.3.7-6.el8.noarch.rpm 336 kB/s | 126 kB 00:00 2024-05-08T19:57:10.464 INFO:teuthology.orchestra.run.smithi188.stdout:(8/14): python3-requests-oauthlib-1.0.0-1.el8.n 268 kB/s | 43 kB 00:00 2024-05-08T19:57:10.573 INFO:teuthology.orchestra.run.smithi188.stdout:(9/14): python3-cachetools-3.1.1-4.el8.noarch.r 164 kB/s | 33 kB 00:00 2024-05-08T19:57:10.598 INFO:teuthology.orchestra.run.smithi188.stdout:(10/14): python3-certifi-2018.10.15-7.el8.noarc 81 kB/s | 16 kB 00:00 2024-05-08T19:57:10.698 INFO:teuthology.orchestra.run.smithi188.stdout:(11/14): python3-google-auth-1.1.1-10.el8.noarc 347 kB/s | 81 kB 00:00 2024-05-08T19:57:10.724 INFO:teuthology.orchestra.run.smithi188.stdout:(12/14): python3-rsa-4.9-2.el8.noarch.rpm 512 kB/s | 64 kB 00:00 2024-05-08T19:57:10.774 INFO:teuthology.orchestra.run.smithi188.stdout:(13/14): python3-websocket-client-0.56.0-5.el8. 814 kB/s | 61 kB 00:00 2024-05-08T19:57:10.833 INFO:teuthology.orchestra.run.smithi188.stdout:(14/14): python3-kubernetes-11.0.0-6.el8.noarch 4.5 MB/s | 1.2 MB 00:00 2024-05-08T19:57:10.835 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:57:10.835 INFO:teuthology.orchestra.run.smithi188.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-05-08T19:57:10.893 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-05-08T19:57:10.908 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-05-08T19:57:10.908 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-05-08T19:57:10.995 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-05-08T19:57:10.998 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-05-08T19:57:11.553 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-05-08T19:57:11.832 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-05-08T19:57:11.836 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/14 2024-05-08T19:57:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/14 2024-05-08T19:57:12.262 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/14 2024-05-08T19:57:12.494 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/14 2024-05-08T19:57:12.684 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/14 2024-05-08T19:57:12.940 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/14 2024-05-08T19:57:13.046 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-05-08T19:57:13.200 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/14 2024-05-08T19:57:13.250 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1322.gd21e4f9 8/8 2024-05-08T19:57:13.457 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 8/14 2024-05-08T19:57:13.572 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1322.gd21e4f9 8/8 2024-05-08T19:57:13.573 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1322.gd21e4f9 1/8 2024-05-08T19:57:13.573 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-05-08T19:57:13.573 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-05-08T19:57:13.573 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-05-08T19:57:13.573 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-05-08T19:57:13.573 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-05-08T19:57:13.573 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libgfortran-8.5.0-22.el8.x86_64 7/8 2024-05-08T19:57:13.724 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 9/14 2024-05-08T19:57:13.988 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libquadmath-8.5.0-22.el8.x86_64 8/8 2024-05-08T19:57:13.988 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:13.988 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-05-08T19:57:13.988 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T19:57:13.988 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran-8.5.0-22.el8.x86_64 2024-05-08T19:57:13.988 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath-8.5.0-22.el8.x86_64 2024-05-08T19:57:13.988 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-0.3.15-4.el8.x86_64 2024-05-08T19:57:13.988 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-05-08T19:57:13.988 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-05-08T19:57:13.988 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-05-08T19:57:13.988 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-05-08T19:57:13.988 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:13.989 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:57:14.117 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-rook 2024-05-08T19:57:14.746 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 10/14 2024-05-08T19:57:14.881 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:03 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:57:15.002 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 11/14 2024-05-08T19:57:15.237 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 12/14 2024-05-08T19:57:15.456 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 13/14 2024-05-08T19:57:15.686 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-rook-2:18.2.2-1322.gd21e4f96.el8.noarch 14/14 2024-05-08T19:57:15.981 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1322.gd21e4f96.el8.noarch 14/14 2024-05-08T19:57:15.982 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1322.gd21e4f96.el8.noarch 1/14 2024-05-08T19:57:15.982 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/14 2024-05-08T19:57:15.982 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/14 2024-05-08T19:57:15.982 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/14 2024-05-08T19:57:15.982 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/14 2024-05-08T19:57:15.982 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/14 2024-05-08T19:57:15.982 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 7/14 2024-05-08T19:57:15.982 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/14 2024-05-08T19:57:15.982 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 9/14 2024-05-08T19:57:15.982 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 10/14 2024-05-08T19:57:15.982 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 11/14 2024-05-08T19:57:15.982 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 12/14 2024-05-08T19:57:15.982 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 13/14 2024-05-08T19:57:16.529 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:57:16.531 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:57:16.531 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-05-08T19:57:16.531 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:57:16.531 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-05-08T19:57:16.531 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 49 k 2024-05-08T19:57:16.531 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-05-08T19:57:16.531 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-05-08T19:57:16.531 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-05-08T19:57:16.531 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-05-08T19:57:16.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-05-08T19:57:16.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-05-08T19:57:16.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-05-08T19:57:16.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-05-08T19:57:16.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-05-08T19:57:16.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-05-08T19:57:16.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-05-08T19:57:16.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-05-08T19:57:16.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-05-08T19:57:16.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-05-08T19:57:16.532 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:16.532 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-05-08T19:57:16.533 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:57:16.533 INFO:teuthology.orchestra.run.smithi083.stdout:Install 14 Packages 2024-05-08T19:57:16.533 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:16.533 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 2.0 M 2024-05-08T19:57:16.533 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 25 M 2024-05-08T19:57:16.533 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-05-08T19:57:16.774 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 14/14 2024-05-08T19:57:16.774 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:16.774 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-05-08T19:57:16.774 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T19:57:16.774 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-05-08T19:57:16.774 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-05-08T19:57:16.774 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-05-08T19:57:16.774 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-05-08T19:57:16.774 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-05-08T19:57:16.774 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-05-08T19:57:16.775 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-05-08T19:57:16.775 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-05-08T19:57:16.775 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-05-08T19:57:16.775 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-05-08T19:57:16.775 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-05-08T19:57:16.775 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa-4.9-2.el8.noarch 2024-05-08T19:57:16.775 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-05-08T19:57:16.775 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:16.775 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:57:16.873 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-cephadm 2024-05-08T19:57:17.059 INFO:teuthology.orchestra.run.smithi083.stdout:(1/14): ceph-mgr-rook-18.2.2-1322.gd21e4f96.el8 293 kB/s | 49 kB 00:00 2024-05-08T19:57:17.117 INFO:teuthology.orchestra.run.smithi083.stdout:(2/14): python3-jsonpointer-1.10-11.el8.noarch. 88 kB/s | 20 kB 00:00 2024-05-08T19:57:17.142 INFO:teuthology.orchestra.run.smithi083.stdout:(3/14): python3-jsonpatch-1.21-2.el8.noarch.rpm 105 kB/s | 27 kB 00:00 2024-05-08T19:57:17.326 INFO:teuthology.orchestra.run.smithi083.stdout:(4/14): python3-jwt-1.6.1-2.el8.noarch.rpm 232 kB/s | 43 kB 00:00 2024-05-08T19:57:17.352 INFO:teuthology.orchestra.run.smithi083.stdout:(5/14): python3-pyasn1-modules-0.3.7-6.el8.noar 470 kB/s | 110 kB 00:00 2024-05-08T19:57:17.435 INFO:teuthology.orchestra.run.smithi083.stdout:(6/14): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.4 MB/s | 155 kB 00:00 2024-05-08T19:57:17.460 INFO:teuthology.orchestra.run.smithi083.stdout:(7/14): python3-pyasn1-0.3.7-6.el8.noarch.rpm 315 kB/s | 126 kB 00:00 2024-05-08T19:57:17.528 INFO:teuthology.orchestra.run.smithi083.stdout:(8/14): python3-requests-oauthlib-1.0.0-1.el8.n 243 kB/s | 43 kB 00:00 2024-05-08T19:57:17.634 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:17 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:57:17.645 INFO:teuthology.orchestra.run.smithi083.stdout:(9/14): python3-certifi-2018.10.15-7.el8.noarch 89 kB/s | 16 kB 00:00 2024-05-08T19:57:17.670 INFO:teuthology.orchestra.run.smithi083.stdout:(10/14): python3-cachetools-3.1.1-4.el8.noarch. 141 kB/s | 33 kB 00:00 2024-05-08T19:57:17.728 INFO:teuthology.orchestra.run.smithi083.stdout:(11/14): python3-google-auth-1.1.1-10.el8.noarc 405 kB/s | 81 kB 00:00 2024-05-08T19:57:17.762 INFO:teuthology.orchestra.run.smithi083.stdout:(12/14): python3-rsa-4.9-2.el8.noarch.rpm 698 kB/s | 64 kB 00:00 2024-05-08T19:57:17.787 INFO:teuthology.orchestra.run.smithi083.stdout:(13/14): python3-websocket-client-0.56.0-5.el8. 1.0 MB/s | 61 kB 00:00 2024-05-08T19:57:18.204 INFO:teuthology.orchestra.run.smithi083.stdout:(14/14): python3-kubernetes-11.0.0-6.el8.noarch 2.1 MB/s | 1.2 MB 00:00 2024-05-08T19:57:18.207 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:57:18.207 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.2 MB/s | 2.0 MB 00:01 2024-05-08T19:57:18.273 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-05-08T19:57:18.289 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-05-08T19:57:18.289 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-05-08T19:57:18.376 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-05-08T19:57:18.379 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-05-08T19:57:18.908 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-05-08T19:57:19.179 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/14 2024-05-08T19:57:19.269 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:57:19.270 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:57:19.271 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-05-08T19:57:19.271 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:57:19.271 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-05-08T19:57:19.271 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 138 k 2024-05-08T19:57:19.271 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-05-08T19:57:19.271 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-05-08T19:57:19.271 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-05-08T19:57:19.271 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-05-08T19:57:19.271 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-05-08T19:57:19.271 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:19.271 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-05-08T19:57:19.271 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:57:19.271 INFO:teuthology.orchestra.run.smithi188.stdout:Install 5 Packages 2024-05-08T19:57:19.271 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:19.272 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 6.0 M 2024-05-08T19:57:19.272 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 26 M 2024-05-08T19:57:19.272 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-05-08T19:57:19.357 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/14 2024-05-08T19:57:19.576 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/14 2024-05-08T19:57:19.814 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/14 2024-05-08T19:57:20.039 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/14 2024-05-08T19:57:20.271 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/14 2024-05-08T19:57:20.526 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/14 2024-05-08T19:57:20.780 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 8/14 2024-05-08T19:57:20.934 INFO:teuthology.orchestra.run.smithi188.stdout:(1/5): ceph-mgr-cephadm-18.2.2-1322.gd21e4f96.e 345 kB/s | 138 kB 00:00 2024-05-08T19:57:21.038 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 9/14 2024-05-08T19:57:21.042 INFO:teuthology.orchestra.run.smithi188.stdout:(2/5): python3-jinja2-2.10.1-4.el8.noarch.rpm 1.1 MB/s | 578 kB 00:00 2024-05-08T19:57:21.494 INFO:teuthology.orchestra.run.smithi188.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 5.0 MB/s | 4.8 MB 00:00 2024-05-08T19:57:21.543 INFO:teuthology.orchestra.run.smithi188.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 120 kB/s | 60 kB 00:00 2024-05-08T19:57:21.701 INFO:teuthology.orchestra.run.smithi188.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 620 kB/s | 476 kB 00:00 2024-05-08T19:57:21.703 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:57:21.703 INFO:teuthology.orchestra.run.smithi188.stdout:Total 2.5 MB/s | 6.0 MB 00:02 2024-05-08T19:57:21.757 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-05-08T19:57:21.769 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-05-08T19:57:21.769 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-05-08T19:57:21.878 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-05-08T19:57:21.882 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-05-08T19:57:22.046 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 10/14 2024-05-08T19:57:22.156 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-05-08T19:57:22.280 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 11/14 2024-05-08T19:57:22.515 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2024-05-08T19:57:22.516 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 12/14 2024-05-08T19:57:22.768 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 13/14 2024-05-08T19:57:22.966 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-rook-2:18.2.2-1322.gd21e4f96.el8.noarch 14/14 2024-05-08T19:57:23.272 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1322.gd21e4f96.el8.noarch 14/14 2024-05-08T19:57:23.272 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1322.gd21e4f96.el8.noarch 1/14 2024-05-08T19:57:23.273 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/14 2024-05-08T19:57:23.273 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/14 2024-05-08T19:57:23.273 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/14 2024-05-08T19:57:23.273 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/14 2024-05-08T19:57:23.273 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/14 2024-05-08T19:57:23.273 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 7/14 2024-05-08T19:57:23.273 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/14 2024-05-08T19:57:23.273 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 9/14 2024-05-08T19:57:23.273 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 10/14 2024-05-08T19:57:23.273 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 11/14 2024-05-08T19:57:23.273 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 12/14 2024-05-08T19:57:23.273 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 13/14 2024-05-08T19:57:23.442 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-05-08T19:57:23.770 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 14/14 2024-05-08T19:57:23.770 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:23.771 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-05-08T19:57:23.771 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T19:57:23.771 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-05-08T19:57:23.771 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-05-08T19:57:23.771 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-05-08T19:57:23.771 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-05-08T19:57:23.771 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-05-08T19:57:23.771 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-05-08T19:57:23.771 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-05-08T19:57:23.771 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-05-08T19:57:23.771 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-05-08T19:57:23.771 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-05-08T19:57:23.771 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-05-08T19:57:23.771 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa-4.9-2.el8.noarch 2024-05-08T19:57:23.771 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-05-08T19:57:23.772 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:23.772 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:57:23.796 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2024-05-08T19:57:23.878 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-cephadm 2024-05-08T19:57:24.093 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 4/5 2024-05-08T19:57:24.343 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 5/5 2024-05-08T19:57:24.630 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 5/5 2024-05-08T19:57:24.630 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 1/5 2024-05-08T19:57:24.631 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2024-05-08T19:57:24.631 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/5 2024-05-08T19:57:24.631 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-05-08T19:57:24.631 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:13 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:57:25.002 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2024-05-08T19:57:25.002 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:25.002 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-05-08T19:57:25.002 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T19:57:25.002 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-05-08T19:57:25.002 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel-2.5.1-7.el8.noarch 2024-05-08T19:57:25.003 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-05-08T19:57:25.003 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-05-08T19:57:25.003 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:25.003 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:57:25.119 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-fuse 2024-05-08T19:57:25.872 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:25 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:57:26.276 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:57:26.277 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:57:26.278 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-05-08T19:57:26.278 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:57:26.278 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-05-08T19:57:26.278 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 138 k 2024-05-08T19:57:26.278 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-05-08T19:57:26.278 INFO:teuthology.orchestra.run.smithi083.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-05-08T19:57:26.279 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-05-08T19:57:26.279 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-05-08T19:57:26.279 INFO:teuthology.orchestra.run.smithi083.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-05-08T19:57:26.279 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:26.279 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-05-08T19:57:26.279 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:57:26.280 INFO:teuthology.orchestra.run.smithi083.stdout:Install 5 Packages 2024-05-08T19:57:26.280 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:26.281 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 6.0 M 2024-05-08T19:57:26.281 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 26 M 2024-05-08T19:57:26.281 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-05-08T19:57:26.730 INFO:teuthology.orchestra.run.smithi083.stdout:(1/5): ceph-mgr-cephadm-18.2.2-1322.gd21e4f96.e 407 kB/s | 138 kB 00:00 2024-05-08T19:57:26.897 INFO:teuthology.orchestra.run.smithi083.stdout:(2/5): python3-jinja2-2.10.1-4.el8.noarch.rpm 1.1 MB/s | 578 kB 00:00 2024-05-08T19:57:27.030 INFO:teuthology.orchestra.run.smithi083.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 452 kB/s | 60 kB 00:00 2024-05-08T19:57:27.080 INFO:teuthology.orchestra.run.smithi083.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.3 MB/s | 476 kB 00:00 2024-05-08T19:57:27.239 INFO:teuthology.orchestra.run.smithi083.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 5.6 MB/s | 4.8 MB 00:00 2024-05-08T19:57:27.240 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:57:27.240 INFO:teuthology.orchestra.run.smithi083.stdout:Total 6.2 MB/s | 6.0 MB 00:00 2024-05-08T19:57:27.292 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-05-08T19:57:27.303 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-05-08T19:57:27.303 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-05-08T19:57:27.405 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-05-08T19:57:27.408 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-05-08T19:57:27.524 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:57:27.524 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:57:27.524 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-05-08T19:57:27.525 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:57:27.525 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-05-08T19:57:27.525 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 860 k 2024-05-08T19:57:27.525 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-05-08T19:57:27.525 INFO:teuthology.orchestra.run.smithi188.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-05-08T19:57:27.525 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:27.525 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-05-08T19:57:27.525 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:57:27.525 INFO:teuthology.orchestra.run.smithi188.stdout:Install 2 Packages 2024-05-08T19:57:27.525 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:27.526 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 943 k 2024-05-08T19:57:27.526 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 3.0 M 2024-05-08T19:57:27.526 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-05-08T19:57:27.689 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-05-08T19:57:27.902 INFO:teuthology.orchestra.run.smithi188.stdout:(1/2): fuse-2.9.7-19.el8.x86_64.rpm 431 kB/s | 83 kB 00:00 2024-05-08T19:57:27.952 INFO:teuthology.orchestra.run.smithi188.stdout:(2/2): ceph-fuse-18.2.2-1322.gd21e4f96.el8.x86_ 3.5 MB/s | 860 kB 00:00 2024-05-08T19:57:27.953 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:57:27.953 INFO:teuthology.orchestra.run.smithi188.stdout:Total 2.2 MB/s | 943 kB 00:00 2024-05-08T19:57:27.963 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-05-08T19:57:27.985 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-05-08T19:57:27.985 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-05-08T19:57:28.037 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2024-05-08T19:57:28.071 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-05-08T19:57:28.074 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-05-08T19:57:28.253 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-05-08T19:57:28.519 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-05-08T19:57:28.734 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 2/2 2024-05-08T19:57:28.971 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-05-08T19:57:29.128 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 2/2 2024-05-08T19:57:29.129 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/2 2024-05-08T19:57:29.317 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2024-05-08T19:57:29.524 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-05-08T19:57:29.525 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:29.525 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-05-08T19:57:29.525 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-05-08T19:57:29.525 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:29.525 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:57:29.607 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 4/5 2024-05-08T19:57:29.739 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-volume 2024-05-08T19:57:29.832 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 5/5 2024-05-08T19:57:30.116 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 5/5 2024-05-08T19:57:30.117 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 1/5 2024-05-08T19:57:30.117 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2024-05-08T19:57:30.117 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/5 2024-05-08T19:57:30.117 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-05-08T19:57:30.493 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2024-05-08T19:57:30.493 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:30.493 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-05-08T19:57:30.494 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T19:57:30.494 INFO:teuthology.orchestra.run.smithi083.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-05-08T19:57:30.494 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel-2.5.1-7.el8.noarch 2024-05-08T19:57:30.494 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-05-08T19:57:30.494 INFO:teuthology.orchestra.run.smithi083.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-05-08T19:57:30.494 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:30.494 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:57:30.505 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:30 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:57:30.615 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-fuse 2024-05-08T19:57:31.363 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:20 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:57:32.124 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:57:32.124 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:57:32.124 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-05-08T19:57:32.125 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:57:32.125 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-05-08T19:57:32.125 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 263 k 2024-05-08T19:57:32.125 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-05-08T19:57:32.125 INFO:teuthology.orchestra.run.smithi188.stdout: python3-packaging noarch 16.8-9.el8 powertools 48 k 2024-05-08T19:57:32.125 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:32.125 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-05-08T19:57:32.125 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:57:32.125 INFO:teuthology.orchestra.run.smithi188.stdout:Install 2 Packages 2024-05-08T19:57:32.125 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:32.126 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 311 k 2024-05-08T19:57:32.126 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 1.4 M 2024-05-08T19:57:32.126 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-05-08T19:57:32.801 INFO:teuthology.orchestra.run.smithi188.stdout:(1/2): ceph-volume-18.2.2-1322.gd21e4f96.el8.no 464 kB/s | 263 kB 00:00 2024-05-08T19:57:33.018 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:57:33.019 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:57:33.019 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-05-08T19:57:33.019 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:57:33.019 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-05-08T19:57:33.019 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 860 k 2024-05-08T19:57:33.019 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-05-08T19:57:33.019 INFO:teuthology.orchestra.run.smithi083.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-05-08T19:57:33.019 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:33.019 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-05-08T19:57:33.019 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:57:33.019 INFO:teuthology.orchestra.run.smithi083.stdout:Install 2 Packages 2024-05-08T19:57:33.019 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:33.020 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 943 k 2024-05-08T19:57:33.020 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 3.0 M 2024-05-08T19:57:33.020 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-05-08T19:57:33.110 INFO:teuthology.orchestra.run.smithi188.stdout:(2/2): python3-packaging-16.8-9.el8.noarch.rpm 55 kB/s | 48 kB 00:00 2024-05-08T19:57:33.111 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:57:33.111 INFO:teuthology.orchestra.run.smithi188.stdout:Total 315 kB/s | 311 kB 00:00 2024-05-08T19:57:33.123 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-05-08T19:57:33.136 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-05-08T19:57:33.136 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-05-08T19:57:33.182 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-05-08T19:57:33.185 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-05-08T19:57:33.371 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-05-08T19:57:33.506 INFO:teuthology.orchestra.run.smithi083.stdout:(1/2): fuse-2.9.7-19.el8.x86_64.rpm 317 kB/s | 83 kB 00:00 2024-05-08T19:57:33.697 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-packaging-16.8-9.el8.noarch 1/2 2024-05-08T19:57:33.974 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-volume-2:18.2.2-1322.gd21e4f96.el8.noarch 2/2 2024-05-08T19:57:34.384 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-volume-2:18.2.2-1322.gd21e4f96.el8.noarch 2/2 2024-05-08T19:57:34.384 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-volume-2:18.2.2-1322.gd21e4f96.el8.noarch 1/2 2024-05-08T19:57:34.763 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 2/2 2024-05-08T19:57:34.763 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:34.763 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-05-08T19:57:34.763 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T19:57:34.763 INFO:teuthology.orchestra.run.smithi188.stdout: python3-packaging-16.8-9.el8.noarch 2024-05-08T19:57:34.763 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:34.764 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:57:35.020 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install librados-devel 2024-05-08T19:57:35.432 INFO:teuthology.orchestra.run.smithi083.stdout:(2/2): ceph-fuse-18.2.2-1322.gd21e4f96.el8.x86_ 393 kB/s | 860 kB 00:02 2024-05-08T19:57:35.433 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:57:35.433 INFO:teuthology.orchestra.run.smithi083.stdout:Total 391 kB/s | 943 kB 00:02 2024-05-08T19:57:35.445 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-05-08T19:57:35.468 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-05-08T19:57:35.468 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-05-08T19:57:35.560 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-05-08T19:57:35.563 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-05-08T19:57:35.729 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-05-08T19:57:35.785 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:35 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:57:36.037 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-05-08T19:57:36.239 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 2/2 2024-05-08T19:57:36.635 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 2/2 2024-05-08T19:57:36.635 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/2 2024-05-08T19:57:37.014 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-05-08T19:57:37.014 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:37.014 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-05-08T19:57:37.014 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-05-08T19:57:37.014 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:37.014 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:57:37.250 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-volume 2024-05-08T19:57:37.437 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:57:37.437 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:57:37.437 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-05-08T19:57:37.437 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:57:37.437 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-05-08T19:57:37.438 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 139 k 2024-05-08T19:57:37.438 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:37.438 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-05-08T19:57:37.438 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:57:37.438 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-05-08T19:57:37.438 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:37.438 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 139 k 2024-05-08T19:57:37.439 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 513 k 2024-05-08T19:57:37.439 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-05-08T19:57:37.825 INFO:teuthology.orchestra.run.smithi188.stdout:librados-devel-18.2.2-1322.gd21e4f96.el8.x86_64 360 kB/s | 139 kB 00:00 2024-05-08T19:57:37.826 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:57:37.826 INFO:teuthology.orchestra.run.smithi188.stdout:Total 358 kB/s | 139 kB 00:00 2024-05-08T19:57:37.826 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-05-08T19:57:37.832 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-05-08T19:57:37.832 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-05-08T19:57:37.881 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-05-08T19:57:37.884 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-05-08T19:57:38.004 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-05-08T19:57:38.013 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:27 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:57:38.222 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librados-devel-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:57:38.501 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados-devel-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:57:39.459 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados-devel-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:57:39.459 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:39.459 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-05-08T19:57:39.459 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:57:39.459 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:39.459 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:57:39.685 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:57:39.686 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:57:39.686 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-05-08T19:57:39.686 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:57:39.686 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-05-08T19:57:39.686 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-volume noarch 2:18.2.2-1322.gd21e4f96.el8 ceph-noarch 263 k 2024-05-08T19:57:39.687 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-05-08T19:57:39.687 INFO:teuthology.orchestra.run.smithi083.stdout: python3-packaging noarch 16.8-9.el8 powertools 48 k 2024-05-08T19:57:39.687 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:39.687 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-05-08T19:57:39.687 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:57:39.687 INFO:teuthology.orchestra.run.smithi083.stdout:Install 2 Packages 2024-05-08T19:57:39.687 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:39.688 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 311 k 2024-05-08T19:57:39.688 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 1.4 M 2024-05-08T19:57:39.688 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-05-08T19:57:39.989 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install libcephfs2 2024-05-08T19:57:40.078 INFO:teuthology.orchestra.run.smithi083.stdout:(1/2): python3-packaging-16.8-9.el8.noarch.rpm 266 kB/s | 48 kB 00:00 2024-05-08T19:57:40.453 INFO:teuthology.orchestra.run.smithi083.stdout:(2/2): ceph-volume-18.2.2-1322.gd21e4f96.el8.no 473 kB/s | 263 kB 00:00 2024-05-08T19:57:40.453 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:57:40.454 INFO:teuthology.orchestra.run.smithi083.stdout:Total 406 kB/s | 311 kB 00:00 2024-05-08T19:57:40.464 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-05-08T19:57:40.476 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-05-08T19:57:40.476 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-05-08T19:57:40.517 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-05-08T19:57:40.520 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-05-08T19:57:40.670 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-05-08T19:57:40.766 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:40 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:57:40.985 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-packaging-16.8-9.el8.noarch 1/2 2024-05-08T19:57:41.272 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-volume-2:18.2.2-1322.gd21e4f96.el8.noarch 2/2 2024-05-08T19:57:41.664 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-volume-2:18.2.2-1322.gd21e4f96.el8.noarch 2/2 2024-05-08T19:57:41.665 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-volume-2:18.2.2-1322.gd21e4f96.el8.noarch 1/2 2024-05-08T19:57:42.075 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 2/2 2024-05-08T19:57:42.075 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:42.075 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-05-08T19:57:42.076 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-volume-2:18.2.2-1322.gd21e4f96.el8.noarch 2024-05-08T19:57:42.076 INFO:teuthology.orchestra.run.smithi083.stdout: python3-packaging-16.8-9.el8.noarch 2024-05-08T19:57:42.076 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:42.076 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:57:42.286 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados-devel 2024-05-08T19:57:42.340 INFO:teuthology.orchestra.run.smithi188.stdout:Package libcephfs2-2:18.2.2-1322.gd21e4f96.el8.x86_64 is already installed. 2024-05-08T19:57:42.387 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:57:42.389 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-05-08T19:57:42.389 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:57:42.487 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install libcephfs-devel 2024-05-08T19:57:43.040 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:32 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:57:43.238 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:43 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:57:44.699 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:57:44.700 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:57:44.700 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-05-08T19:57:44.700 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:57:44.700 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-05-08T19:57:44.700 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 139 k 2024-05-08T19:57:44.700 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:44.700 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-05-08T19:57:44.700 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:57:44.700 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-05-08T19:57:44.700 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:44.700 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 139 k 2024-05-08T19:57:44.700 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 513 k 2024-05-08T19:57:44.701 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-05-08T19:57:44.887 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:57:44.887 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:57:44.887 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-05-08T19:57:44.887 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:57:44.887 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-05-08T19:57:44.887 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 31 k 2024-05-08T19:57:44.887 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:44.888 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-05-08T19:57:44.888 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:57:44.888 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-05-08T19:57:44.888 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:44.888 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 31 k 2024-05-08T19:57:44.889 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 139 k 2024-05-08T19:57:44.889 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-05-08T19:57:44.897 INFO:teuthology.orchestra.run.smithi083.stdout:librados-devel-18.2.2-1322.gd21e4f96.el8.x86_64 706 kB/s | 139 kB 00:00 2024-05-08T19:57:44.898 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:57:44.898 INFO:teuthology.orchestra.run.smithi083.stdout:Total 701 kB/s | 139 kB 00:00 2024-05-08T19:57:44.898 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-05-08T19:57:44.904 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-05-08T19:57:44.904 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-05-08T19:57:44.952 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-05-08T19:57:44.955 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-05-08T19:57:45.063 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-05-08T19:57:45.278 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librados-devel-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:57:45.325 INFO:teuthology.orchestra.run.smithi188.stdout:libcephfs-devel-18.2.2-1322.gd21e4f96.el8.x86_6 71 kB/s | 31 kB 00:00 2024-05-08T19:57:45.326 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:57:45.326 INFO:teuthology.orchestra.run.smithi188.stdout:Total 70 kB/s | 31 kB 00:00 2024-05-08T19:57:45.327 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-05-08T19:57:45.331 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-05-08T19:57:45.331 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-05-08T19:57:45.367 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-05-08T19:57:45.370 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-05-08T19:57:45.472 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-05-08T19:57:45.554 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados-devel-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:57:45.687 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs-devel-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:57:45.924 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados-devel-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:57:45.924 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:45.924 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-05-08T19:57:45.924 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:57:45.924 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:45.925 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:57:45.949 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:57:46.152 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install libcephfs2 2024-05-08T19:57:46.290 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs-devel-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:57:46.290 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:46.290 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-05-08T19:57:46.290 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:57:46.290 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:57:46.290 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:57:46.423 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install librados2 2024-05-08T19:57:46.913 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:35 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:57:47.171 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:47 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:57:48.518 INFO:teuthology.orchestra.run.smithi083.stdout:Package libcephfs2-2:18.2.2-1322.gd21e4f96.el8.x86_64 is already installed. 2024-05-08T19:57:48.568 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:57:48.570 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-05-08T19:57:48.570 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:57:48.681 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install libcephfs-devel 2024-05-08T19:57:48.781 INFO:teuthology.orchestra.run.smithi188.stdout:Package librados2-2:18.2.2-1322.gd21e4f96.el8.x86_64 is already installed. 2024-05-08T19:57:48.829 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:57:48.830 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-05-08T19:57:48.830 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:57:48.930 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install librbd1 2024-05-08T19:57:49.429 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:38 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:57:49.677 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:49 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:57:51.081 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:57:51.081 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:57:51.081 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-05-08T19:57:51.081 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:57:51.081 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-05-08T19:57:51.081 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 31 k 2024-05-08T19:57:51.081 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:51.082 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-05-08T19:57:51.082 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:57:51.082 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-05-08T19:57:51.082 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:51.082 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 31 k 2024-05-08T19:57:51.082 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 139 k 2024-05-08T19:57:51.082 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-05-08T19:57:51.275 INFO:teuthology.orchestra.run.smithi188.stdout:Package librbd1-2:18.2.2-1322.gd21e4f96.el8.x86_64 is already installed. 2024-05-08T19:57:51.310 INFO:teuthology.orchestra.run.smithi083.stdout:libcephfs-devel-18.2.2-1322.gd21e4f96.el8.x86_6 136 kB/s | 31 kB 00:00 2024-05-08T19:57:51.310 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:57:51.310 INFO:teuthology.orchestra.run.smithi083.stdout:Total 135 kB/s | 31 kB 00:00 2024-05-08T19:57:51.311 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-05-08T19:57:51.314 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-05-08T19:57:51.314 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-05-08T19:57:51.323 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:57:51.324 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-05-08T19:57:51.324 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:57:51.340 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-05-08T19:57:51.343 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-05-08T19:57:51.406 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-05-08T19:57:51.427 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-rados 2024-05-08T19:57:51.644 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs-devel-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:57:51.908 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:57:52.180 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:52 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:57:52.254 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs-devel-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:57:52.254 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:52.254 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-05-08T19:57:52.254 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:57:52.254 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:57:52.255 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:57:52.370 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados2 2024-05-08T19:57:53.112 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:42 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:57:53.788 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-rados-2:18.2.2-1322.gd21e4f96.el8.x86_64 is already installed. 2024-05-08T19:57:53.836 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:57:53.837 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-05-08T19:57:53.837 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:57:53.952 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-rgw 2024-05-08T19:57:54.701 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:54 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:57:54.707 INFO:teuthology.orchestra.run.smithi083.stdout:Package librados2-2:18.2.2-1322.gd21e4f96.el8.x86_64 is already installed. 2024-05-08T19:57:54.754 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:57:54.755 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-05-08T19:57:54.755 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:57:54.860 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librbd1 2024-05-08T19:57:55.603 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:44 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:57:56.314 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-rgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 is already installed. 2024-05-08T19:57:56.363 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:57:56.364 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-05-08T19:57:56.364 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:57:56.462 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-cephfs 2024-05-08T19:57:57.205 INFO:teuthology.orchestra.run.smithi083.stdout:Package librbd1-2:18.2.2-1322.gd21e4f96.el8.x86_64 is already installed. 2024-05-08T19:57:57.225 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:57 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:57:57.253 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:57:57.263 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-05-08T19:57:57.263 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:57:57.362 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rados 2024-05-08T19:57:58.113 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:47 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:57:58.817 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-cephfs-2:18.2.2-1322.gd21e4f96.el8.x86_64 is already installed. 2024-05-08T19:57:58.864 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:57:58.865 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-05-08T19:57:58.865 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:57:58.974 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-rbd 2024-05-08T19:57:59.700 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rados-2:18.2.2-1322.gd21e4f96.el8.x86_64 is already installed. 2024-05-08T19:57:59.725 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:59 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:57:59.747 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:57:59.749 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-05-08T19:57:59.749 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:57:59.851 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rgw 2024-05-08T19:58:00.645 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:49 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:58:01.311 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-rbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 is already installed. 2024-05-08T19:58:01.359 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:58:01.360 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-05-08T19:58:01.360 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:58:01.471 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-fuse 2024-05-08T19:58:02.214 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:03:02 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:58:02.246 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rgw-2:18.2.2-1322.gd21e4f96.el8.x86_64 is already installed. 2024-05-08T19:58:02.294 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:58:02.295 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-05-08T19:58:02.296 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:58:02.403 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-cephfs 2024-05-08T19:58:03.149 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:52 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:58:03.833 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:58:03.834 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:58:03.834 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-05-08T19:58:03.834 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:58:03.834 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-05-08T19:58:03.834 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 95 k 2024-05-08T19:58:03.834 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:58:03.834 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-05-08T19:58:03.834 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:58:03.834 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-05-08T19:58:03.834 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:58:03.835 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 95 k 2024-05-08T19:58:03.835 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 280 k 2024-05-08T19:58:03.835 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-05-08T19:58:04.129 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-fuse-18.2.2-1322.gd21e4f96.el8.x86_64.rpm 324 kB/s | 95 kB 00:00 2024-05-08T19:58:04.129 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:58:04.129 INFO:teuthology.orchestra.run.smithi188.stdout:Total 323 kB/s | 95 kB 00:00 2024-05-08T19:58:04.130 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-05-08T19:58:04.140 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-05-08T19:58:04.140 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-05-08T19:58:04.188 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-05-08T19:58:04.191 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-05-08T19:58:04.291 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-05-08T19:58:04.474 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:58:04.753 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-cephfs-2:18.2.2-1322.gd21e4f96.el8.x86_64 is already installed. 2024-05-08T19:58:04.762 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:58:04.801 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:58:04.802 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-05-08T19:58:04.802 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:58:04.907 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rbd 2024-05-08T19:58:05.150 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:58:05.150 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:58:05.150 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-05-08T19:58:05.151 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:58:05.151 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:58:05.151 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:58:05.419 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-mirror 2024-05-08T19:58:05.657 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:54 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:58:06.183 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:03:06 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:58:07.258 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 is already installed. 2024-05-08T19:58:07.306 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:58:07.307 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-05-08T19:58:07.307 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:58:07.408 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-fuse 2024-05-08T19:58:07.834 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:58:07.835 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:58:07.835 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-05-08T19:58:07.835 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:58:07.835 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-05-08T19:58:07.835 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 3.5 M 2024-05-08T19:58:07.835 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:58:07.835 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-05-08T19:58:07.835 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:58:07.836 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-05-08T19:58:07.836 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:58:07.836 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 3.5 M 2024-05-08T19:58:07.836 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 16 M 2024-05-08T19:58:07.836 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-05-08T19:58:08.162 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:57 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:58:08.457 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-mirror-18.2.2-1322.gd21e4f96.el8.x86_64.rpm 5.6 MB/s | 3.5 MB 00:00 2024-05-08T19:58:08.458 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:58:08.458 INFO:teuthology.orchestra.run.smithi188.stdout:Total 5.5 MB/s | 3.5 MB 00:00 2024-05-08T19:58:08.458 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-05-08T19:58:08.471 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-05-08T19:58:08.471 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-05-08T19:58:08.535 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-05-08T19:58:08.538 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-05-08T19:58:08.998 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-05-08T19:58:09.182 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:58:09.594 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:58:09.815 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:58:09.815 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:58:09.815 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-05-08T19:58:09.815 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:58:09.815 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-05-08T19:58:09.815 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 95 k 2024-05-08T19:58:09.815 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:58:09.815 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-05-08T19:58:09.816 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:58:09.816 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-05-08T19:58:09.816 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:58:09.816 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 95 k 2024-05-08T19:58:09.816 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 280 k 2024-05-08T19:58:09.816 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-05-08T19:58:09.968 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:58:09.968 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:58:09.968 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-05-08T19:58:09.968 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:58:09.969 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:58:09.969 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:58:10.000 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-fuse-18.2.2-1322.gd21e4f96.el8.x86_64.rpm 514 kB/s | 95 kB 00:00 2024-05-08T19:58:10.000 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:58:10.000 INFO:teuthology.orchestra.run.smithi083.stdout:Total 510 kB/s | 95 kB 00:00 2024-05-08T19:58:10.001 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-05-08T19:58:10.011 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-05-08T19:58:10.011 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-05-08T19:58:10.060 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-05-08T19:58:10.063 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-05-08T19:58:10.165 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-05-08T19:58:10.173 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-nbd 2024-05-08T19:58:10.333 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:58:10.623 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:58:10.933 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:03:10 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T19:58:11.009 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:58:11.009 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:58:11.009 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-05-08T19:58:11.009 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:58:11.009 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:58:11.009 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:58:11.244 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-mirror 2024-05-08T19:58:12.009 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:03:01 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:58:12.591 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T19:58:12.592 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:58:12.592 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-05-08T19:58:12.592 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:58:12.592 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-05-08T19:58:12.592 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 181 k 2024-05-08T19:58:12.592 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:58:12.592 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-05-08T19:58:12.592 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T19:58:12.592 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-05-08T19:58:12.592 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:58:12.593 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 181 k 2024-05-08T19:58:12.593 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 572 k 2024-05-08T19:58:12.593 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-05-08T19:58:12.886 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-nbd-18.2.2-1322.gd21e4f96.el8.x86_64.rpm 619 kB/s | 181 kB 00:00 2024-05-08T19:58:12.886 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:58:12.886 INFO:teuthology.orchestra.run.smithi188.stdout:Total 617 kB/s | 181 kB 00:00 2024-05-08T19:58:12.887 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-05-08T19:58:12.898 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-05-08T19:58:12.898 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-05-08T19:58:12.948 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-05-08T19:58:12.951 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-05-08T19:58:13.083 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-05-08T19:58:13.264 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-nbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:58:13.597 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:58:13.658 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:58:13.658 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:58:13.658 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-05-08T19:58:13.658 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:58:13.658 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-05-08T19:58:13.658 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 3.5 M 2024-05-08T19:58:13.658 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:58:13.659 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-05-08T19:58:13.659 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:58:13.659 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-05-08T19:58:13.659 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:58:13.659 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 3.5 M 2024-05-08T19:58:13.659 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 16 M 2024-05-08T19:58:13.659 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-05-08T19:58:13.994 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-nbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:58:13.995 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:58:13.995 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-05-08T19:58:13.995 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:58:13.995 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T19:58:13.995 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T19:58:14.195 DEBUG:teuthology.parallel:result is None 2024-05-08T19:58:14.603 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-mirror-18.2.2-1322.gd21e4f96.el8.x86_64.rpm 3.7 MB/s | 3.5 MB 00:00 2024-05-08T19:58:14.604 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:58:14.604 INFO:teuthology.orchestra.run.smithi083.stdout:Total 3.7 MB/s | 3.5 MB 00:00 2024-05-08T19:58:14.604 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-05-08T19:58:14.617 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-05-08T19:58:14.617 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-05-08T19:58:14.680 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-05-08T19:58:14.683 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-05-08T19:58:15.119 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-05-08T19:58:15.301 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:58:15.719 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:58:16.110 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:58:16.110 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:58:16.110 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-05-08T19:58:16.110 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:58:16.110 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:58:16.110 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:58:16.356 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-nbd 2024-05-08T19:58:17.111 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:03:06 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T19:58:18.751 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T19:58:18.752 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:58:18.752 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-05-08T19:58:18.752 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:58:18.752 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-05-08T19:58:18.752 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd x86_64 2:18.2.2-1322.gd21e4f96.el8 ceph 181 k 2024-05-08T19:58:18.752 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:58:18.752 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-05-08T19:58:18.752 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T19:58:18.752 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-05-08T19:58:18.752 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:58:18.753 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 181 k 2024-05-08T19:58:18.753 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 572 k 2024-05-08T19:58:18.753 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-05-08T19:58:18.932 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-nbd-18.2.2-1322.gd21e4f96.el8.x86_64.rpm 1.0 MB/s | 181 kB 00:00 2024-05-08T19:58:18.932 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:58:18.933 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.0 MB/s | 181 kB 00:00 2024-05-08T19:58:18.933 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-05-08T19:58:18.944 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-05-08T19:58:18.944 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-05-08T19:58:18.992 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-05-08T19:58:18.995 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-05-08T19:58:19.139 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-05-08T19:58:19.307 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-nbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:58:19.592 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:58:19.970 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-nbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 1/1 2024-05-08T19:58:19.970 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:58:19.970 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-05-08T19:58:19.970 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd-2:18.2.2-1322.gd21e4f96.el8.x86_64 2024-05-08T19:58:19.970 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:58:19.970 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T19:58:20.206 DEBUG:teuthology.parallel:result is None 2024-05-08T19:58:20.207 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d21e4f961318b56f0d915caa2490c3711bf13c34 2024-05-08T19:58:20.528 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-08T19:58:20.551 INFO:teuthology.orchestra.run.smithi083.stdout:18.2.2-1322.gd21e4f96.el8 2024-05-08T19:58:20.551 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1322.gd21e4f96.el8 2024-05-08T19:58:20.551 INFO:teuthology.task.install:The correct ceph version 18.2.2-1322.gd21e4f96 is installed. 2024-05-08T19:58:20.553 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d21e4f961318b56f0d915caa2490c3711bf13c34 2024-05-08T19:58:20.845 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-08T19:58:20.871 INFO:teuthology.orchestra.run.smithi188.stdout:18.2.2-1322.gd21e4f96.el8 2024-05-08T19:58:20.872 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1322.gd21e4f96.el8 2024-05-08T19:58:20.872 INFO:teuthology.task.install:The correct ceph version 18.2.2-1322.gd21e4f96 is installed. 2024-05-08T19:58:20.873 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-08T19:58:20.874 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-05-08T19:58:20.874 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-08T19:58:20.901 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-05-08T19:58:20.901 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-08T19:58:20.944 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-08T19:58:20.944 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-05-08T19:58:20.944 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/daemon-helper 2024-05-08T19:58:20.971 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-08T19:58:21.038 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-05-08T19:58:21.038 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/daemon-helper 2024-05-08T19:58:21.067 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-08T19:58:21.137 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-08T19:58:21.137 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-05-08T19:58:21.137 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-08T19:58:21.163 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-08T19:58:21.230 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-05-08T19:58:21.230 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-08T19:58:21.259 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-08T19:58:21.328 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-08T19:58:21.328 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-05-08T19:58:21.328 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/stdin-killer 2024-05-08T19:58:21.355 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-08T19:58:21.421 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-05-08T19:58:21.421 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/stdin-killer 2024-05-08T19:58:21.449 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-08T19:58:21.517 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-08T19:58:21.614 INFO:tasks.cephadm:Config: {'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'd21e4f961318b56f0d915caa2490c3711bf13c34'} 2024-05-08T19:58:21.614 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 2024-05-08T19:58:21.614 INFO:tasks.cephadm:Cluster fsid is 526038a6-0d75-11ef-bc97-c7b262605968 2024-05-08T19:58:21.614 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-08T19:58:21.615 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-05-08T19:58:21.615 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi083': '172.21.15.83', 'mon.smithi188': '172.21.15.188'} 2024-05-08T19:58:21.615 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-08T19:58:21.615 DEBUG:teuthology.orchestra.run.smithi083:> sudo hostname $(hostname -s) 2024-05-08T19:58:21.643 DEBUG:teuthology.orchestra.run.smithi188:> sudo hostname $(hostname -s) 2024-05-08T19:58:21.672 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-05-08T19:58:21.672 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d21e4f961318b56f0d915caa2490c3711bf13c34 2024-05-08T19:58:21.811 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd21e4f961318b56f0d915caa2490c3711bf13c34', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/79222/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.2.2-1322-gd21e4f96', 'node_name': '172.21.2.4+braggi04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.2.2-1322.gd21e4f96'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.150423-reef/d21e4f961318b56f0d915caa2490c3711bf13c34/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2024-05-08 19:11:17.581213', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20240508.150423-reef', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20240508.150423-reef/d21e4f961318b56f0d915caa2490c3711bf13c34/centos/8/flavors/default/', 'archs': ['arm64', 'source', 'x86_64'], 'distro': 'centos'}, {'status': 'ready', 'sha1': 'd21e4f961318b56f0d915caa2490c3711bf13c34', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/79220/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.2.2-1322-gd21e4f96', 'node_name': '172.21.2.7+braggi07', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.2.2-1322.gd21e4f96'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.150423-reef/d21e4f961318b56f0d915caa2490c3711bf13c34/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2024-05-08 18:26:58.979379', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20240508.150423-reef', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20240508.150423-reef/d21e4f961318b56f0d915caa2490c3711bf13c34/centos/8/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-05-08T19:58:21.951 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-pdonnell-testing-20240508.150423-reef, sha1 d21e4f961318b56f0d915caa2490c3711bf13c34 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=d21e4f961318b56f0d915caa2490c3711bf13c34 2024-05-08T19:58:21.953 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240508.150423-reef/d21e4f961318b56f0d915caa2490c3711bf13c34/centos/8/x86_64/flavors/default/cephadm 2024-05-08T19:58:21.953 DEBUG:teuthology.orchestra.run.smithi083:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240508.150423-reef/d21e4f961318b56f0d915caa2490c3711bf13c34/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-08T19:58:22.706 INFO:teuthology.orchestra.run.smithi083.stdout:-rw-rw-r--. 1 ubuntu ubuntu 703022 May 8 19:58 /home/ubuntu/cephtest/cephadm 2024-05-08T19:58:22.707 DEBUG:teuthology.orchestra.run.smithi188:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240508.150423-reef/d21e4f961318b56f0d915caa2490c3711bf13c34/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-08T19:58:23.085 INFO:teuthology.orchestra.run.smithi188.stdout:-rw-rw-r--. 1 ubuntu ubuntu 703022 May 8 19:58 /home/ubuntu/cephtest/cephadm 2024-05-08T19:58:23.086 DEBUG:teuthology.orchestra.run.smithi083:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-08T19:58:23.108 DEBUG:teuthology.orchestra.run.smithi188:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-08T19:58:23.140 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 on all hosts... 2024-05-08T19:58:23.141 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 pull 2024-05-08T19:58:23.151 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 pull 2024-05-08T19:58:24.301 INFO:teuthology.orchestra.run.smithi083.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34... 2024-05-08T19:58:24.404 INFO:teuthology.orchestra.run.smithi188.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34... 2024-05-08T19:58:56.306 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-05-08T19:58:56.307 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_version": "ceph version 18.2.2-1322-gd21e4f96 (d21e4f961318b56f0d915caa2490c3711bf13c34) reef (stable)", 2024-05-08T19:58:56.307 INFO:teuthology.orchestra.run.smithi083.stdout: "image_id": "af23193ff691dd58bee78ea954b569105e41c41def20b81a25baa1d30500ad1c", 2024-05-08T19:58:56.307 INFO:teuthology.orchestra.run.smithi083.stdout: "repo_digests": [ 2024-05-08T19:58:56.307 INFO:teuthology.orchestra.run.smithi083.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e8ec72da4fcdc307fe9b94614a1c07d7ebae386ad20c4998c8d80e9b80a48985" 2024-05-08T19:58:56.307 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-05-08T19:58:56.307 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-05-08T19:58:57.340 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-05-08T19:58:57.340 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_version": "ceph version 18.2.2-1322-gd21e4f96 (d21e4f961318b56f0d915caa2490c3711bf13c34) reef (stable)", 2024-05-08T19:58:57.340 INFO:teuthology.orchestra.run.smithi188.stdout: "image_id": "af23193ff691dd58bee78ea954b569105e41c41def20b81a25baa1d30500ad1c", 2024-05-08T19:58:57.341 INFO:teuthology.orchestra.run.smithi188.stdout: "repo_digests": [ 2024-05-08T19:58:57.341 INFO:teuthology.orchestra.run.smithi188.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e8ec72da4fcdc307fe9b94614a1c07d7ebae386ad20c4998c8d80e9b80a48985" 2024-05-08T19:58:57.341 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-05-08T19:58:57.341 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-05-08T19:58:57.361 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph 2024-05-08T19:58:57.396 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /etc/ceph 2024-05-08T19:58:57.427 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 777 /etc/ceph 2024-05-08T19:58:57.467 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 777 /etc/ceph 2024-05-08T19:58:57.493 INFO:tasks.cephadm:Writing seed config... 2024-05-08T19:58:57.494 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-08T19:58:57.494 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-08T19:58:57.494 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-08T19:58:57.494 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-08T19:58:57.494 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-08T19:58:57.494 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-08T19:58:57.495 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-08T19:58:57.495 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-05-08T19:58:57.495 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-05-08T19:58:57.495 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-08T19:58:57.527 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 526038a6-0d75-11ef-bc97-c7b262605968 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-05-08T19:58:57.527 DEBUG:teuthology.orchestra.run.smithi083:mon.smithi083> sudo journalctl -f -n 0 -u ceph-526038a6-0d75-11ef-bc97-c7b262605968@mon.smithi083.service 2024-05-08T19:58:57.570 INFO:tasks.cephadm:Bootstrapping... 2024-05-08T19:58:57.570 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 -v bootstrap --fsid 526038a6-0d75-11ef-bc97-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.83 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-08T19:58:57.599 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:-- Logs begin at Wed 2024-05-08 19:47:49 UTC. -- 2024-05-08T19:58:57.736 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-05-08T19:58:57.736 INFO:teuthology.orchestra.run.smithi083.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34', '-v', 'bootstrap', '--fsid', '526038a6-0d75-11ef-bc97-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.83', '--skip-admin-label'] 2024-05-08T19:58:57.768 INFO:teuthology.orchestra.run.smithi083.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-08T19:58:57.768 INFO:teuthology.orchestra.run.smithi083.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-05-08T19:58:57.769 INFO:teuthology.orchestra.run.smithi083.stdout:Verifying podman|docker is present... 2024-05-08T19:58:57.803 INFO:teuthology.orchestra.run.smithi083.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-08T19:58:57.803 INFO:teuthology.orchestra.run.smithi083.stdout:Verifying lvm2 is present... 2024-05-08T19:58:57.803 INFO:teuthology.orchestra.run.smithi083.stdout:Verifying time synchronization is in place... 2024-05-08T19:58:57.810 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-08T19:58:57.810 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-08T19:58:57.816 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-08T19:58:57.816 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout inactive 2024-05-08T19:58:57.822 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout enabled 2024-05-08T19:58:57.827 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout active 2024-05-08T19:58:57.828 INFO:teuthology.orchestra.run.smithi083.stdout:Unit chronyd.service is enabled and running 2024-05-08T19:58:57.828 INFO:teuthology.orchestra.run.smithi083.stdout:Repeating the final host check... 2024-05-08T19:58:57.855 INFO:teuthology.orchestra.run.smithi083.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-08T19:58:57.855 INFO:teuthology.orchestra.run.smithi083.stdout:podman (/bin/podman) version 4.9.4 is present 2024-05-08T19:58:57.855 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl is present 2024-05-08T19:58:57.856 INFO:teuthology.orchestra.run.smithi083.stdout:lvcreate is present 2024-05-08T19:58:57.861 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-08T19:58:57.862 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-08T19:58:57.867 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-08T19:58:57.867 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout inactive 2024-05-08T19:58:57.872 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout enabled 2024-05-08T19:58:57.879 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout active 2024-05-08T19:58:57.879 INFO:teuthology.orchestra.run.smithi083.stdout:Unit chronyd.service is enabled and running 2024-05-08T19:58:57.879 INFO:teuthology.orchestra.run.smithi083.stdout:Host looks OK 2024-05-08T19:58:57.879 INFO:teuthology.orchestra.run.smithi083.stdout:Cluster fsid: 526038a6-0d75-11ef-bc97-c7b262605968 2024-05-08T19:58:57.879 INFO:teuthology.orchestra.run.smithi083.stdout:Acquiring lock 140566157195752 on /run/cephadm/526038a6-0d75-11ef-bc97-c7b262605968.lock 2024-05-08T19:58:57.879 INFO:teuthology.orchestra.run.smithi083.stdout:Lock 140566157195752 acquired on /run/cephadm/526038a6-0d75-11ef-bc97-c7b262605968.lock 2024-05-08T19:58:57.879 INFO:teuthology.orchestra.run.smithi083.stdout:Verifying IP 172.21.15.83 port 3300 ... 2024-05-08T19:58:57.880 INFO:teuthology.orchestra.run.smithi083.stdout:Verifying IP 172.21.15.83 port 6789 ... 2024-05-08T19:58:57.880 INFO:teuthology.orchestra.run.smithi083.stdout:Base mon IP(s) is [172.21.15.83:3300, 172.21.15.83:6789], mon addrv is [v2:172.21.15.83:3300,v1:172.21.15.83:6789] 2024-05-08T19:58:57.883 INFO:teuthology.orchestra.run.smithi083.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.83 metric 100 2024-05-08T19:58:57.883 INFO:teuthology.orchestra.run.smithi083.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.83 metric 100 2024-05-08T19:58:57.886 INFO:teuthology.orchestra.run.smithi083.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-05-08T19:58:57.886 INFO:teuthology.orchestra.run.smithi083.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-05-08T19:58:57.886 INFO:teuthology.orchestra.run.smithi083.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1795sec hoplimit 64 pref medium 2024-05-08T19:58:57.888 INFO:teuthology.orchestra.run.smithi083.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-08T19:58:57.888 INFO:teuthology.orchestra.run.smithi083.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-05-08T19:58:57.888 INFO:teuthology.orchestra.run.smithi083.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-08T19:58:57.889 INFO:teuthology.orchestra.run.smithi083.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-05-08T19:58:57.889 INFO:teuthology.orchestra.run.smithi083.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:ce91/64 scope link 2024-05-08T19:58:57.889 INFO:teuthology.orchestra.run.smithi083.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-08T19:58:57.889 INFO:teuthology.orchestra.run.smithi083.stdout:Mon IP `172.21.15.83` is in CIDR network `172.21.0.0/20` 2024-05-08T19:58:57.889 INFO:teuthology.orchestra.run.smithi083.stdout:Mon IP `172.21.15.83` is in CIDR network `172.21.0.0/20` 2024-05-08T19:58:57.889 INFO:teuthology.orchestra.run.smithi083.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-05-08T19:58:57.890 INFO:teuthology.orchestra.run.smithi083.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-08T19:58:57.890 INFO:teuthology.orchestra.run.smithi083.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34... 2024-05-08T19:58:58.268 INFO:teuthology.orchestra.run.smithi083.stdout:/bin/podman: stdout af23193ff691dd58bee78ea954b569105e41c41def20b81a25baa1d30500ad1c 2024-05-08T19:58:58.268 INFO:teuthology.orchestra.run.smithi083.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34... 2024-05-08T19:58:58.268 INFO:teuthology.orchestra.run.smithi083.stdout:/bin/podman: stderr Getting image source signatures 2024-05-08T19:58:58.268 INFO:teuthology.orchestra.run.smithi083.stdout:/bin/podman: stderr Copying blob sha256:7850648fa4707607dddc5eec7674b8c9296086cf87f20a8b1acf1b014b952e9f 2024-05-08T19:58:58.268 INFO:teuthology.orchestra.run.smithi083.stdout:/bin/podman: stderr Copying blob sha256:bf5b95bd68251da39c6cd2eb96c6ba4138bb37a62bdc39287908b25e92047d7e 2024-05-08T19:58:58.268 INFO:teuthology.orchestra.run.smithi083.stdout:/bin/podman: stderr Copying config sha256:af23193ff691dd58bee78ea954b569105e41c41def20b81a25baa1d30500ad1c 2024-05-08T19:58:58.268 INFO:teuthology.orchestra.run.smithi083.stdout:/bin/podman: stderr Writing manifest to image destination 2024-05-08T19:58:59.305 INFO:teuthology.orchestra.run.smithi083.stdout:ceph: stdout ceph version 18.2.2-1322-gd21e4f96 (d21e4f961318b56f0d915caa2490c3711bf13c34) reef (stable) 2024-05-08T19:58:59.306 INFO:teuthology.orchestra.run.smithi083.stdout:Ceph version: ceph version 18.2.2-1322-gd21e4f96 (d21e4f961318b56f0d915caa2490c3711bf13c34) reef (stable) 2024-05-08T19:58:59.306 INFO:teuthology.orchestra.run.smithi083.stdout:Extracting ceph user uid/gid from container image... 2024-05-08T19:59:00.371 INFO:teuthology.orchestra.run.smithi083.stdout:stat: stdout 167 167 2024-05-08T19:59:00.371 INFO:teuthology.orchestra.run.smithi083.stdout:Creating initial keys... 2024-05-08T19:59:01.502 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-authtool: stdout AQCE2TtmlDYULxAAJTMOBr+z3Fj3IHSYi8jy0Q== 2024-05-08T19:59:02.589 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-authtool: stdout AQCF2TtmMDGrNxAAlW7r12/DZlaAwRaPEeRwGQ== 2024-05-08T19:59:03.755 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-authtool: stdout AQCH2Ttml/4dBRAAc+JxNXZztGDGpzwUTUy+7Q== 2024-05-08T19:59:03.756 INFO:teuthology.orchestra.run.smithi083.stdout:Creating initial monmap... 2024-05-08T19:59:04.969 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-08T19:59:04.970 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-05-08T19:59:04.970 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 526038a6-0d75-11ef-bc97-c7b262605968 2024-05-08T19:59:04.970 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-08T19:59:04.970 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool for smithi083 [v2:172.21.15.83:3300,v1:172.21.15.83:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-08T19:59:04.970 INFO:teuthology.orchestra.run.smithi083.stdout:setting min_mon_release = pacific 2024-05-08T19:59:04.970 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/monmaptool: set fsid to 526038a6-0d75-11ef-bc97-c7b262605968 2024-05-08T19:59:04.970 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-08T19:59:04.970 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T19:59:04.970 INFO:teuthology.orchestra.run.smithi083.stdout:Creating mon... 2024-05-08T19:59:06.013 INFO:teuthology.orchestra.run.smithi083.stdout:create mon.smithi083 on 2024-05-08T19:59:06.135 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-05-08T19:59:06.221 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-08T19:59:06.334 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-526038a6-0d75-11ef-bc97-c7b262605968.target → /etc/systemd/system/ceph-526038a6-0d75-11ef-bc97-c7b262605968.target. 2024-05-08T19:59:06.334 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-526038a6-0d75-11ef-bc97-c7b262605968.target → /etc/systemd/system/ceph-526038a6-0d75-11ef-bc97-c7b262605968.target. 2024-05-08T19:59:06.528 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-526038a6-0d75-11ef-bc97-c7b262605968@mon.smithi083 2024-05-08T19:59:06.528 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Failed to reset failed state of unit ceph-526038a6-0d75-11ef-bc97-c7b262605968@mon.smithi083.service: Unit ceph-526038a6-0d75-11ef-bc97-c7b262605968@mon.smithi083.service not loaded. 2024-05-08T19:59:06.619 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-526038a6-0d75-11ef-bc97-c7b262605968.target.wants/ceph-526038a6-0d75-11ef-bc97-c7b262605968@mon.smithi083.service → /etc/systemd/system/ceph-526038a6-0d75-11ef-bc97-c7b262605968@.service. 2024-05-08T19:59:07.194 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-08T19:59:07.194 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout disabled 2024-05-08T19:59:07.202 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-08T19:59:07.202 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout inactive 2024-05-08T19:59:07.202 INFO:teuthology.orchestra.run.smithi083.stdout:firewalld.service is not enabled 2024-05-08T19:59:07.202 INFO:teuthology.orchestra.run.smithi083.stdout:Not possible to enable service . firewalld.service is not available 2024-05-08T19:59:07.202 INFO:teuthology.orchestra.run.smithi083.stdout:Waiting for mon to start... 2024-05-08T19:59:07.203 INFO:teuthology.orchestra.run.smithi083.stdout:Waiting for mon... 2024-05-08T19:59:07.408 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:07 smithi083 bash[38170]: c032a966db2e833b3c505bb7254f9716ace3f5494d6f11946fd359b3f3abbd41 2024-05-08T19:59:07.408 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:07 smithi083 systemd[1]: Started Ceph mon.smithi083 for 526038a6-0d75-11ef-bc97-c7b262605968. 2024-05-08T19:59:08.324 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout cluster: 2024-05-08T19:59:08.324 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout id: 526038a6-0d75-11ef-bc97-c7b262605968 2024-05-08T19:59:08.324 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-05-08T19:59:08.324 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-05-08T19:59:08.324 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout services: 2024-05-08T19:59:08.324 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi083 (age 0.691844s) 2024-05-08T19:59:08.325 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-05-08T19:59:08.325 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-05-08T19:59:08.325 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-05-08T19:59:08.325 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout data: 2024-05-08T19:59:08.325 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-05-08T19:59:08.325 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-05-08T19:59:08.325 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-05-08T19:59:08.325 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout pgs: 2024-05-08T19:59:08.325 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-05-08T19:59:08.325 INFO:teuthology.orchestra.run.smithi083.stdout:mon is available 2024-05-08T19:59:08.325 INFO:teuthology.orchestra.run.smithi083.stdout:Assimilating anything we can from ceph.conf... 2024-05-08T19:59:08.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:08 smithi083 ceph-mon[38208]: mon.smithi083 is new leader, mons smithi083 in quorum (ranks 0) 2024-05-08T19:59:08.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:08 smithi083 ceph-mon[38208]: monmap e1: 1 mons at {smithi083=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-08T19:59:08.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:08 smithi083 ceph-mon[38208]: fsmap 2024-05-08T19:59:08.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:08 smithi083 ceph-mon[38208]: osdmap e1: 0 total, 0 up, 0 in 2024-05-08T19:59:08.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:08 smithi083 ceph-mon[38208]: mgrmap e1: no daemons active 2024-05-08T19:59:08.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:08 smithi083 ceph-mon[38208]: from='client.? 172.21.15.83:0/1124390301' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-08T19:59:09.343 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:09 smithi083 ceph-mon[38208]: from='client.? 172.21.15.83:0/3421366539' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-08T19:59:09.343 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:09 smithi083 ceph-mon[38208]: from='client.? 172.21.15.83:0/3421366539' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-08T19:59:09.554 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-05-08T19:59:09.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout [global] 2024-05-08T19:59:09.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout fsid = 526038a6-0d75-11ef-bc97-c7b262605968 2024-05-08T19:59:09.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.83:3300,v1:172.21.15.83:6789] 2024-05-08T19:59:09.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-08T19:59:09.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-08T19:59:09.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-08T19:59:09.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-08T19:59:09.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-05-08T19:59:09.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-08T19:59:09.557 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-08T19:59:09.557 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-05-08T19:59:09.557 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout [osd] 2024-05-08T19:59:09.557 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-08T19:59:09.557 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-08T19:59:09.557 INFO:teuthology.orchestra.run.smithi083.stdout:Generating new minimal ceph.conf... 2024-05-08T19:59:10.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:10 smithi083 ceph-mon[38208]: from='client.? 172.21.15.83:0/2299878518' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T19:59:10.769 INFO:teuthology.orchestra.run.smithi083.stdout:Restarting the monitor... 2024-05-08T19:59:11.044 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:10 smithi083 systemd[1]: Stopping Ceph mon.smithi083 for 526038a6-0d75-11ef-bc97-c7b262605968... 2024-05-08T19:59:11.361 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:11 smithi083 ceph-526038a6-0d75-11ef-bc97-c7b262605968-mon-smithi083[38185]: 2024-05-08T19:59:11.053+0000 7f2e627b3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi083 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-08T19:59:11.361 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:11 smithi083 ceph-526038a6-0d75-11ef-bc97-c7b262605968-mon-smithi083[38185]: 2024-05-08T19:59:11.053+0000 7f2e627b3700 -1 mon.smithi083@0(leader) e1 *** Got Signal Terminated *** 2024-05-08T19:59:11.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:11 smithi083 bash[38535]: ceph-526038a6-0d75-11ef-bc97-c7b262605968-mon-smithi083 2024-05-08T19:59:12.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:11 smithi083 systemd[1]: ceph-526038a6-0d75-11ef-bc97-c7b262605968@mon.smithi083.service: Succeeded. 2024-05-08T19:59:12.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:11 smithi083 systemd[1]: Stopped Ceph mon.smithi083 for 526038a6-0d75-11ef-bc97-c7b262605968. 2024-05-08T19:59:12.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:11 smithi083 systemd[1]: Starting Ceph mon.smithi083 for 526038a6-0d75-11ef-bc97-c7b262605968... 2024-05-08T19:59:12.519 INFO:teuthology.orchestra.run.smithi083.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-05-08T19:59:12.719 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: set uid:gid to 167:167 (ceph:ceph) 2024-05-08T19:59:12.719 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: ceph version 18.2.2-1322-gd21e4f96 (d21e4f961318b56f0d915caa2490c3711bf13c34) reef (stable), process ceph-mon, pid 7 2024-05-08T19:59:12.719 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: pidfile_write: ignore empty --pid-file 2024-05-08T19:59:12.719 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: load: jerasure load: lrc 2024-05-08T19:59:12.719 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: RocksDB version: 7.9.2 2024-05-08T19:59:12.719 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Git sha 0 2024-05-08T19:59:12.719 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Compile date 2024-05-08 17:55:50 2024-05-08T19:59:12.720 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: DB SUMMARY 2024-05-08T19:59:12.720 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: DB Session ID: DTYTA8LOVCETMFB01JVC 2024-05-08T19:59:12.720 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: CURRENT file: CURRENT 2024-05-08T19:59:12.720 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: IDENTITY file: IDENTITY 2024-05-08T19:59:12.720 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-05-08T19:59:12.720 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi083/store.db dir, Total Num: 1, files: 000008.sst 2024-05-08T19:59:12.720 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi083/store.db: 000009.log size: 83798 ; 2024-05-08T19:59:12.720 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.error_if_exists: 0 2024-05-08T19:59:12.720 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.create_if_missing: 0 2024-05-08T19:59:12.720 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.paranoid_checks: 1 2024-05-08T19:59:12.720 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-08T19:59:12.720 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-08T19:59:12.720 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-08T19:59:12.720 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.env: 0x556c416b0820 2024-05-08T19:59:12.721 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.fs: PosixFileSystem 2024-05-08T19:59:12.721 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.info_log: 0x556c434d13e0 2024-05-08T19:59:12.721 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_file_opening_threads: 16 2024-05-08T19:59:12.721 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.statistics: (nil) 2024-05-08T19:59:12.721 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.use_fsync: 0 2024-05-08T19:59:12.721 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_log_file_size: 0 2024-05-08T19:59:12.721 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-08T19:59:12.721 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-08T19:59:12.721 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.keep_log_file_num: 1000 2024-05-08T19:59:12.721 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.recycle_log_file_num: 0 2024-05-08T19:59:12.721 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.allow_fallocate: 1 2024-05-08T19:59:12.721 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.allow_mmap_reads: 0 2024-05-08T19:59:12.721 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.allow_mmap_writes: 0 2024-05-08T19:59:12.722 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.use_direct_reads: 0 2024-05-08T19:59:12.722 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-08T19:59:12.722 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.create_missing_column_families: 0 2024-05-08T19:59:12.722 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.db_log_dir: 2024-05-08T19:59:12.722 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.wal_dir: 2024-05-08T19:59:12.722 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-08T19:59:12.722 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-08T19:59:12.722 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-08T19:59:12.722 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-08T19:59:12.722 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-08T19:59:12.722 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-08T19:59:12.722 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.advise_random_on_open: 1 2024-05-08T19:59:12.722 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.db_write_buffer_size: 0 2024-05-08T19:59:12.722 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.write_buffer_manager: 0x556c42746140 2024-05-08T19:59:12.723 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-08T19:59:12.723 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-08T19:59:12.723 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-08T19:59:12.723 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.rate_limiter: (nil) 2024-05-08T19:59:12.723 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-08T19:59:12.723 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.wal_recovery_mode: 2 2024-05-08T19:59:12.723 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.enable_thread_tracking: 0 2024-05-08T19:59:12.723 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.enable_pipelined_write: 0 2024-05-08T19:59:12.723 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.unordered_write: 0 2024-05-08T19:59:12.723 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-08T19:59:12.723 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-08T19:59:12.723 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-08T19:59:12.723 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-08T19:59:12.723 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.row_cache: None 2024-05-08T19:59:12.724 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.wal_filter: None 2024-05-08T19:59:12.724 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-08T19:59:12.724 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.allow_ingest_behind: 0 2024-05-08T19:59:12.724 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.two_write_queues: 0 2024-05-08T19:59:12.724 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.manual_wal_flush: 0 2024-05-08T19:59:12.724 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.wal_compression: 0 2024-05-08T19:59:12.724 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.atomic_flush: 0 2024-05-08T19:59:12.724 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-08T19:59:12.724 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-08T19:59:12.724 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-08T19:59:12.724 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.log_readahead_size: 0 2024-05-08T19:59:12.724 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-08T19:59:12.724 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.best_efforts_recovery: 0 2024-05-08T19:59:12.725 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-08T19:59:12.725 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-08T19:59:12.725 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.allow_data_in_errors: 0 2024-05-08T19:59:12.725 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.db_host_id: __hostname__ 2024-05-08T19:59:12.725 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-08T19:59:12.725 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_background_jobs: 2 2024-05-08T19:59:12.725 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_background_compactions: -1 2024-05-08T19:59:12.725 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_subcompactions: 1 2024-05-08T19:59:12.725 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-08T19:59:12.725 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-08T19:59:12.725 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-08T19:59:12.725 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_total_wal_size: 0 2024-05-08T19:59:12.725 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-08T19:59:12.725 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-08T19:59:12.725 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-08T19:59:12.726 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-08T19:59:12.726 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_open_files: -1 2024-05-08T19:59:12.726 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.bytes_per_sync: 0 2024-05-08T19:59:12.726 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-08T19:59:12.726 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-08T19:59:12.726 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.compaction_readahead_size: 0 2024-05-08T19:59:12.726 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_background_flushes: -1 2024-05-08T19:59:12.726 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Compression algorithms supported: 2024-05-08T19:59:12.726 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-08T19:59:12.726 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: kZSTD supported: 0 2024-05-08T19:59:12.726 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: kXpressCompression supported: 0 2024-05-08T19:59:12.726 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: kLZ4HCCompression supported: 1 2024-05-08T19:59:12.726 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: kZlibCompression supported: 1 2024-05-08T19:59:12.726 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: kSnappyCompression supported: 1 2024-05-08T19:59:12.727 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: kLZ4Compression supported: 1 2024-05-08T19:59:12.727 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: kBZip2Compression supported: 0 2024-05-08T19:59:12.727 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-08T19:59:12.727 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-08T19:59:12.727 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi083/store.db/MANIFEST-000010 2024-05-08T19:59:12.727 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-08T19:59:12.727 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-08T19:59:12.727 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.merge_operator: 2024-05-08T19:59:12.727 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.compaction_filter: None 2024-05-08T19:59:12.727 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.compaction_filter_factory: None 2024-05-08T19:59:12.727 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.sst_partitioner_factory: None 2024-05-08T19:59:12.727 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-08T19:59:12.727 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-08T19:59:12.727 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556c434d1500) 2024-05-08T19:59:12.728 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: cache_index_and_filter_blocks: 1 2024-05-08T19:59:12.728 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-08T19:59:12.728 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-08T19:59:12.728 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: pin_top_level_index_and_filter: 1 2024-05-08T19:59:12.728 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: index_type: 0 2024-05-08T19:59:12.728 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: data_block_index_type: 0 2024-05-08T19:59:12.728 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: index_shortening: 1 2024-05-08T19:59:12.728 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-08T19:59:12.728 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: checksum: 4 2024-05-08T19:59:12.728 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: no_block_cache: 0 2024-05-08T19:59:12.728 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: block_cache: 0x556c427e91f0 2024-05-08T19:59:12.728 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: block_cache_name: BinnedLRUCache 2024-05-08T19:59:12.728 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: block_cache_options: 2024-05-08T19:59:12.728 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: capacity : 536870912 2024-05-08T19:59:12.729 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: num_shard_bits : 4 2024-05-08T19:59:12.729 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: strict_capacity_limit : 0 2024-05-08T19:59:12.729 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: high_pri_pool_ratio: 0.000 2024-05-08T19:59:12.729 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: block_cache_compressed: (nil) 2024-05-08T19:59:12.729 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: persistent_cache: (nil) 2024-05-08T19:59:12.729 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: block_size: 4096 2024-05-08T19:59:12.729 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: block_size_deviation: 10 2024-05-08T19:59:12.729 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: block_restart_interval: 16 2024-05-08T19:59:12.729 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: index_block_restart_interval: 1 2024-05-08T19:59:12.729 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: metadata_block_size: 4096 2024-05-08T19:59:12.729 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: partition_filters: 0 2024-05-08T19:59:12.730 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: use_delta_encoding: 1 2024-05-08T19:59:12.730 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: filter_policy: bloomfilter 2024-05-08T19:59:12.730 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: whole_key_filtering: 1 2024-05-08T19:59:12.730 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: verify_compression: 0 2024-05-08T19:59:12.730 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: read_amp_bytes_per_bit: 0 2024-05-08T19:59:12.730 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: format_version: 5 2024-05-08T19:59:12.730 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: enable_index_compression: 1 2024-05-08T19:59:12.730 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: block_align: 0 2024-05-08T19:59:12.730 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: max_auto_readahead_size: 262144 2024-05-08T19:59:12.730 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: prepopulate_block_cache: 0 2024-05-08T19:59:12.730 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: initial_auto_readahead_size: 8192 2024-05-08T19:59:12.730 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: num_file_reads_for_auto_readahead: 2 2024-05-08T19:59:12.730 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.write_buffer_size: 33554432 2024-05-08T19:59:12.730 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_write_buffer_number: 2 2024-05-08T19:59:12.731 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.compression: NoCompression 2024-05-08T19:59:12.731 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.bottommost_compression: Disabled 2024-05-08T19:59:12.731 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.prefix_extractor: nullptr 2024-05-08T19:59:12.731 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-08T19:59:12.731 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.num_levels: 7 2024-05-08T19:59:12.731 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-08T19:59:12.731 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-08T19:59:12.731 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-08T19:59:12.731 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-08T19:59:12.731 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-08T19:59:12.731 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-08T19:59:12.731 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-08T19:59:12.731 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-08T19:59:12.732 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-08T19:59:12.732 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-08T19:59:12.732 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-08T19:59:12.732 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-08T19:59:12.732 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-08T19:59:12.732 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.compression_opts.level: 32767 2024-05-08T19:59:12.732 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.compression_opts.strategy: 0 2024-05-08T19:59:12.732 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-08T19:59:12.732 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-08T19:59:12.732 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-08T19:59:12.733 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-08T19:59:12.733 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.compression_opts.enabled: false 2024-05-08T19:59:12.733 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-08T19:59:12.733 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-08T19:59:12.733 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-08T19:59:12.733 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-08T19:59:12.733 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.target_file_size_base: 67108864 2024-05-08T19:59:12.733 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-08T19:59:12.733 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-08T19:59:12.733 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-08T19:59:12.733 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-08T19:59:12.733 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-08T19:59:12.733 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-08T19:59:12.734 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-08T19:59:12.734 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-08T19:59:12.734 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-08T19:59:12.734 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-08T19:59:12.734 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-08T19:59:12.734 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-08T19:59:12.734 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-08T19:59:12.734 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-08T19:59:12.734 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.arena_block_size: 1048576 2024-05-08T19:59:12.734 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-08T19:59:12.734 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-08T19:59:12.734 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.disable_auto_compactions: 0 2024-05-08T19:59:12.734 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-08T19:59:12.734 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-08T19:59:12.735 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-08T19:59:12.735 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-08T19:59:12.735 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-08T19:59:12.735 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-08T19:59:12.735 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-08T19:59:12.735 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-08T19:59:12.735 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-08T19:59:12.735 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-08T19:59:12.735 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.table_properties_collectors: 2024-05-08T19:59:12.735 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.inplace_update_support: 0 2024-05-08T19:59:12.735 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-08T19:59:12.735 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-08T19:59:12.735 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-08T19:59:12.735 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-08T19:59:12.736 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.bloom_locality: 0 2024-05-08T19:59:12.736 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.max_successive_merges: 0 2024-05-08T19:59:12.736 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-08T19:59:12.736 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.paranoid_file_checks: 0 2024-05-08T19:59:12.736 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.force_consistency_checks: 1 2024-05-08T19:59:12.736 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.report_bg_io_stats: 0 2024-05-08T19:59:12.736 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.ttl: 2592000 2024-05-08T19:59:12.736 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-08T19:59:12.736 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-08T19:59:12.736 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-08T19:59:12.736 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.enable_blob_files: false 2024-05-08T19:59:12.736 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.min_blob_size: 0 2024-05-08T19:59:12.736 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.blob_file_size: 268435456 2024-05-08T19:59:12.736 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-08T19:59:12.736 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-08T19:59:12.737 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-08T19:59:12.737 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-08T19:59:12.737 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-08T19:59:12.737 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.blob_file_starting_level: 0 2024-05-08T19:59:12.737 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-08T19:59:12.737 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi083/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-05-08T19:59:12.737 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-05-08T19:59:12.737 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 40780c38-2c76-41e1-b314-a7749b500e53 2024-05-08T19:59:12.737 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715198352469931, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-05-08T19:59:12.737 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-05-08T19:59:12.737 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715198352470800, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 80570, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 248, "table_properties": {"data_size": 78753, "index_size": 227, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9753, "raw_average_key_size": 47, "raw_value_size": 73173, "raw_average_value_size": 353, "num_data_blocks": 10, "num_entries": 207, "num_filter_entries": 207, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715198352, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "40780c38-2c76-41e1-b314-a7749b500e53", "db_session_id": "DTYTA8LOVCETMFB01JVC", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-05-08T19:59:12.737 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715198352470878, "job": 1, "event": "recovery_finished"} 2024-05-08T19:59:12.737 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-05-08T19:59:12.738 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi083/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-08T19:59:12.738 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556c42886000 2024-05-08T19:59:12.738 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: DB pointer 0x556c42874000 2024-05-08T19:59:12.738 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-08T19:59:12.738 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-08T19:59:12.738 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: ** DB Stats ** 2024-05-08T19:59:12.738 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-08T19:59:12.738 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-08T19:59:12.738 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-08T19:59:12.739 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-08T19:59:12.739 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-08T19:59:12.739 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-08T19:59:12.739 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-08T19:59:12.739 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: 2024-05-08T19:59:12.739 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: ** Compaction Stats [default] ** 2024-05-08T19:59:12.739 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-08T19:59:12.739 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-08T19:59:12.739 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: L0 2/0 80.55 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-08T19:59:12.740 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Sum 2/0 80.55 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-08T19:59:12.740 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-08T19:59:12.740 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: 2024-05-08T19:59:12.740 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: ** Compaction Stats [default] ** 2024-05-08T19:59:12.740 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-08T19:59:12.740 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-08T19:59:12.740 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 137.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-08T19:59:12.740 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: 2024-05-08T19:59:12.740 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-08T19:59:12.740 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: 2024-05-08T19:59:12.740 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-08T19:59:12.740 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-08T19:59:12.740 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-08T19:59:12.740 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-08T19:59:12.741 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-08T19:59:12.741 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-08T19:59:12.741 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Cumulative compaction: 0.00 GB write, 26.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-08T19:59:12.741 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Interval compaction: 0.00 GB write, 26.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-08T19:59:12.741 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-08T19:59:12.741 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Block cache BinnedLRUCache@0x556c427e91f0#7 capacity: 512.00 MB usage: 1.19 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-05-08T19:59:12.741 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.42 KB,8.04663e-05%) Misc(1,0.00 KB,0%) 2024-05-08T19:59:12.741 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: 2024-05-08T19:59:12.741 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-08T19:59:12.741 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: starting mon.smithi083 rank 0 at public addrs [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] at bind addrs [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi083 fsid 526038a6-0d75-11ef-bc97-c7b262605968 2024-05-08T19:59:12.741 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: mon.smithi083@-1(???) e1 preinit fsid 526038a6-0d75-11ef-bc97-c7b262605968 2024-05-08T19:59:12.741 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: mon.smithi083@-1(???).mds e1 new map 2024-05-08T19:59:12.741 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: mon.smithi083@-1(???).mds e1 print_map 2024-05-08T19:59:12.741 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: e1 2024-05-08T19:59:12.742 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-08T19:59:12.742 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-08T19:59:12.742 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: legacy client fscid: -1 2024-05-08T19:59:12.742 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: 2024-05-08T19:59:12.742 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout: No filesystems configured 2024-05-08T19:59:12.742 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: mon.smithi083@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-08T19:59:12.742 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: mon.smithi083@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-08T19:59:12.742 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: mon.smithi083@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-08T19:59:12.742 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: mon.smithi083@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-08T19:59:12.742 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: mon.smithi083@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-08T19:59:12.743 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 bash[38657]: 7b4a52f78f65006cc4d755d0ffe89de8598ded2e2d262d5a4c0b7b05722b0c36 2024-05-08T19:59:12.743 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 systemd[1]: Started Ceph mon.smithi083 for 526038a6-0d75-11ef-bc97-c7b262605968. 2024-05-08T19:59:12.743 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: mon.smithi083 is new leader, mons smithi083 in quorum (ranks 0) 2024-05-08T19:59:12.743 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: monmap e1: 1 mons at {smithi083=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-08T19:59:12.743 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: fsmap 2024-05-08T19:59:12.743 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: osdmap e1: 0 total, 0 up, 0 in 2024-05-08T19:59:12.743 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:12 smithi083 ceph-mon[38695]: mgrmap e1: no daemons active 2024-05-08T19:59:13.599 INFO:teuthology.orchestra.run.smithi083.stdout:Wrote config to /etc/ceph/ceph.conf 2024-05-08T19:59:13.613 INFO:teuthology.orchestra.run.smithi083.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-08T19:59:13.613 INFO:teuthology.orchestra.run.smithi083.stdout:Creating mgr... 2024-05-08T19:59:13.613 INFO:teuthology.orchestra.run.smithi083.stdout:Verifying port 0.0.0.0:9283 ... 2024-05-08T19:59:13.614 INFO:teuthology.orchestra.run.smithi083.stdout:Verifying port 0.0.0.0:8765 ... 2024-05-08T19:59:13.614 INFO:teuthology.orchestra.run.smithi083.stdout:Verifying port 0.0.0.0:8443 ... 2024-05-08T19:59:13.794 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-526038a6-0d75-11ef-bc97-c7b262605968@mgr.smithi083.fgcgxm 2024-05-08T19:59:13.794 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Failed to reset failed state of unit ceph-526038a6-0d75-11ef-bc97-c7b262605968@mgr.smithi083.fgcgxm.service: Unit ceph-526038a6-0d75-11ef-bc97-c7b262605968@mgr.smithi083.fgcgxm.service not loaded. 2024-05-08T19:59:13.882 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-526038a6-0d75-11ef-bc97-c7b262605968.target.wants/ceph-526038a6-0d75-11ef-bc97-c7b262605968@mgr.smithi083.fgcgxm.service → /etc/systemd/system/ceph-526038a6-0d75-11ef-bc97-c7b262605968@.service. 2024-05-08T19:59:14.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:14 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3028908350' entity='client.admin' 2024-05-08T19:59:14.540 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-08T19:59:14.540 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout disabled 2024-05-08T19:59:14.547 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-08T19:59:14.548 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout inactive 2024-05-08T19:59:14.548 INFO:teuthology.orchestra.run.smithi083.stdout:firewalld.service is not enabled 2024-05-08T19:59:14.548 INFO:teuthology.orchestra.run.smithi083.stdout:Not possible to enable service . firewalld.service is not available 2024-05-08T19:59:14.559 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-08T19:59:14.559 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout disabled 2024-05-08T19:59:14.564 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-08T19:59:14.564 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout inactive 2024-05-08T19:59:14.564 INFO:teuthology.orchestra.run.smithi083.stdout:firewalld.service is not enabled 2024-05-08T19:59:14.564 INFO:teuthology.orchestra.run.smithi083.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-05-08T19:59:14.564 INFO:teuthology.orchestra.run.smithi083.stdout:Waiting for mgr to start... 2024-05-08T19:59:14.564 INFO:teuthology.orchestra.run.smithi083.stdout:Waiting for mgr... 2024-05-08T19:59:15.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:15 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/4143757806' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-08T19:59:15.815 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-05-08T19:59:15.815 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout { 2024-05-08T19:59:15.815 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "fsid": "526038a6-0d75-11ef-bc97-c7b262605968", 2024-05-08T19:59:15.815 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "health": { 2024-05-08T19:59:15.815 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-08T19:59:15.815 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-08T19:59:15.815 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-08T19:59:15.816 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-05-08T19:59:15.816 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-08T19:59:15.816 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-08T19:59:15.816 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 0 2024-05-08T19:59:15.816 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout ], 2024-05-08T19:59:15.816 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-08T19:59:15.816 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "smithi083" 2024-05-08T19:59:15.816 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout ], 2024-05-08T19:59:15.816 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-05-08T19:59:15.816 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-08T19:59:15.816 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-08T19:59:15.816 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-05-08T19:59:15.816 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-08T19:59:15.816 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-05-08T19:59:15.817 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-08T19:59:15.817 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-08T19:59:15.817 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-08T19:59:15.817 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-08T19:59:15.817 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-08T19:59:15.817 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-08T19:59:15.817 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-08T19:59:15.817 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-08T19:59:15.817 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-05-08T19:59:15.817 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-08T19:59:15.817 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-08T19:59:15.817 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-08T19:59:15.817 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-08T19:59:15.817 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-08T19:59:15.817 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-08T19:59:15.818 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-08T19:59:15.818 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-08T19:59:15.818 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-08T19:59:15.818 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-05-08T19:59:15.818 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-08T19:59:15.818 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-08T19:59:15.818 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-08T19:59:15.818 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-08T19:59:15.818 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-05-08T19:59:15.818 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-08T19:59:15.818 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-08T19:59:15.818 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-08T19:59:15.818 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-08T19:59:15.819 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-08T19:59:15.819 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-08T19:59:15.819 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "restful" 2024-05-08T19:59:15.819 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout ], 2024-05-08T19:59:15.819 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-08T19:59:15.819 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-05-08T19:59:15.819 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-08T19:59:15.819 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-08T19:59:15.819 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "modified": "2024-05-08T19:59:07.152998+0000", 2024-05-08T19:59:15.819 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-08T19:59:15.819 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-05-08T19:59:15.819 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-08T19:59:15.819 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout } 2024-05-08T19:59:15.819 INFO:teuthology.orchestra.run.smithi083.stdout:mgr not available, waiting (1/15)... 2024-05-08T19:59:21.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:21 smithi083 ceph-mon[38695]: Activating manager daemon smithi083.fgcgxm 2024-05-08T19:59:21.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:21 smithi083 ceph-mon[38695]: mgrmap e2: smithi083.fgcgxm(active, starting, since 0.00279769s) 2024-05-08T19:59:21.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:21 smithi083 ceph-mon[38695]: from='mgr.14100 172.21.15.83:0/1280237102' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-08T19:59:21.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:21 smithi083 ceph-mon[38695]: from='mgr.14100 172.21.15.83:0/1280237102' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-08T19:59:21.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:21 smithi083 ceph-mon[38695]: from='mgr.14100 172.21.15.83:0/1280237102' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-08T19:59:21.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:21 smithi083 ceph-mon[38695]: from='mgr.14100 172.21.15.83:0/1280237102' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi083"}]: dispatch 2024-05-08T19:59:21.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:21 smithi083 ceph-mon[38695]: from='mgr.14100 172.21.15.83:0/1280237102' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mgr metadata", "who": "smithi083.fgcgxm", "id": "smithi083.fgcgxm"}]: dispatch 2024-05-08T19:59:21.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:21 smithi083 ceph-mon[38695]: Manager daemon smithi083.fgcgxm is now available 2024-05-08T19:59:21.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:21 smithi083 ceph-mon[38695]: from='mgr.14100 172.21.15.83:0/1280237102' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.fgcgxm/mirror_snapshot_schedule"}]: dispatch 2024-05-08T19:59:21.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:21 smithi083 ceph-mon[38695]: from='mgr.14100 172.21.15.83:0/1280237102' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.fgcgxm/trash_purge_schedule"}]: dispatch 2024-05-08T19:59:21.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:21 smithi083 ceph-mon[38695]: from='mgr.14100 172.21.15.83:0/1280237102' entity='mgr.smithi083.fgcgxm' 2024-05-08T19:59:21.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:21 smithi083 ceph-mon[38695]: from='mgr.14100 172.21.15.83:0/1280237102' entity='mgr.smithi083.fgcgxm' 2024-05-08T19:59:21.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:21 smithi083 ceph-mon[38695]: from='mgr.14100 172.21.15.83:0/1280237102' entity='mgr.smithi083.fgcgxm' 2024-05-08T19:59:22.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:22 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/119088815' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-08T19:59:22.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:22 smithi083 ceph-mon[38695]: mgrmap e3: smithi083.fgcgxm(active, since 1.00645s) 2024-05-08T19:59:23.549 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-05-08T19:59:23.549 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout { 2024-05-08T19:59:23.549 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "fsid": "526038a6-0d75-11ef-bc97-c7b262605968", 2024-05-08T19:59:23.549 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "health": { 2024-05-08T19:59:23.549 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-08T19:59:23.549 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-08T19:59:23.550 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-08T19:59:23.550 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-05-08T19:59:23.550 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-08T19:59:23.550 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-08T19:59:23.550 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 0 2024-05-08T19:59:23.550 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout ], 2024-05-08T19:59:23.550 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-08T19:59:23.550 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "smithi083" 2024-05-08T19:59:23.550 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout ], 2024-05-08T19:59:23.550 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-05-08T19:59:23.550 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-08T19:59:23.550 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-08T19:59:23.551 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-05-08T19:59:23.551 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-08T19:59:23.551 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-05-08T19:59:23.551 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-08T19:59:23.551 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-08T19:59:23.551 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-08T19:59:23.551 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-08T19:59:23.551 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-08T19:59:23.554 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-08T19:59:23.555 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-08T19:59:23.555 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-08T19:59:23.555 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-05-08T19:59:23.555 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-08T19:59:23.555 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-08T19:59:23.555 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-08T19:59:23.555 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-08T19:59:23.555 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-08T19:59:23.555 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-08T19:59:23.555 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-08T19:59:23.555 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-08T19:59:23.555 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-08T19:59:23.555 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-05-08T19:59:23.555 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-08T19:59:23.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-08T19:59:23.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-08T19:59:23.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-08T19:59:23.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-05-08T19:59:23.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-08T19:59:23.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-08T19:59:23.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-08T19:59:23.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-08T19:59:23.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-08T19:59:23.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-08T19:59:23.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "restful" 2024-05-08T19:59:23.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout ], 2024-05-08T19:59:23.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-08T19:59:23.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-05-08T19:59:23.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-08T19:59:23.556 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-08T19:59:23.557 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "modified": "2024-05-08T19:59:07.152998+0000", 2024-05-08T19:59:23.557 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-08T19:59:23.557 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-05-08T19:59:23.557 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-08T19:59:23.557 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout } 2024-05-08T19:59:23.557 INFO:teuthology.orchestra.run.smithi083.stdout:mgr not available, waiting (2/15)... 2024-05-08T19:59:24.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:24 smithi083 ceph-mon[38695]: mgrmap e4: smithi083.fgcgxm(active, since 2s) 2024-05-08T19:59:26.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:26 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/1804414163' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-08T19:59:26.933 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-05-08T19:59:26.934 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout { 2024-05-08T19:59:26.934 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "fsid": "526038a6-0d75-11ef-bc97-c7b262605968", 2024-05-08T19:59:26.934 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "health": { 2024-05-08T19:59:26.934 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-08T19:59:26.934 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-08T19:59:26.934 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-08T19:59:26.934 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-05-08T19:59:26.934 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-08T19:59:26.934 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-08T19:59:26.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 0 2024-05-08T19:59:26.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout ], 2024-05-08T19:59:26.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-08T19:59:26.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "smithi083" 2024-05-08T19:59:26.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout ], 2024-05-08T19:59:26.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-05-08T19:59:26.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-08T19:59:26.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-08T19:59:26.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-05-08T19:59:26.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-08T19:59:26.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-05-08T19:59:26.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-08T19:59:26.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-08T19:59:26.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-08T19:59:26.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-08T19:59:26.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-08T19:59:26.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-08T19:59:26.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-08T19:59:26.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-08T19:59:26.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-05-08T19:59:26.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-08T19:59:26.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-08T19:59:26.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-08T19:59:26.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-08T19:59:26.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-08T19:59:26.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-08T19:59:26.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-08T19:59:26.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-08T19:59:26.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-08T19:59:26.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-05-08T19:59:26.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-08T19:59:26.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-08T19:59:26.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-08T19:59:26.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-08T19:59:26.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-05-08T19:59:26.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-08T19:59:26.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-08T19:59:26.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-08T19:59:26.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-08T19:59:26.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-08T19:59:26.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-08T19:59:26.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "restful" 2024-05-08T19:59:26.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout ], 2024-05-08T19:59:26.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-08T19:59:26.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-05-08T19:59:26.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-08T19:59:26.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-08T19:59:26.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "modified": "2024-05-08T19:59:07.152998+0000", 2024-05-08T19:59:26.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-08T19:59:26.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-05-08T19:59:26.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-08T19:59:26.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout } 2024-05-08T19:59:26.938 INFO:teuthology.orchestra.run.smithi083.stdout:mgr is available 2024-05-08T19:59:28.022 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:27 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/1546418953' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-08T19:59:28.585 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-05-08T19:59:28.585 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout [global] 2024-05-08T19:59:28.586 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout fsid = 526038a6-0d75-11ef-bc97-c7b262605968 2024-05-08T19:59:28.586 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-08T19:59:28.586 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-08T19:59:28.586 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-08T19:59:28.586 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-08T19:59:28.586 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-05-08T19:59:28.586 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-08T19:59:28.586 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-08T19:59:28.586 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-05-08T19:59:28.586 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout [osd] 2024-05-08T19:59:28.586 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-08T19:59:28.586 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-08T19:59:28.586 INFO:teuthology.orchestra.run.smithi083.stdout:Enabling cephadm module... 2024-05-08T19:59:29.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:29 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3771938277' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-08T19:59:30.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:30 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3771938277' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-08T19:59:30.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:30 smithi083 ceph-mon[38695]: mgrmap e5: smithi083.fgcgxm(active, since 8s) 2024-05-08T19:59:31.473 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout { 2024-05-08T19:59:31.473 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-05-08T19:59:31.473 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-08T19:59:31.473 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "active_name": "smithi083.fgcgxm", 2024-05-08T19:59:31.473 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-08T19:59:31.473 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout } 2024-05-08T19:59:31.473 INFO:teuthology.orchestra.run.smithi083.stdout:Waiting for the mgr to restart... 2024-05-08T19:59:31.473 INFO:teuthology.orchestra.run.smithi083.stdout:Waiting for mgr epoch 5... 2024-05-08T19:59:31.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:31 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2632091372' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-08T19:59:36.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:36 smithi083 ceph-mon[38695]: Active manager daemon smithi083.fgcgxm restarted 2024-05-08T19:59:36.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:36 smithi083 ceph-mon[38695]: Activating manager daemon smithi083.fgcgxm 2024-05-08T19:59:36.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:36 smithi083 ceph-mon[38695]: osdmap e2: 0 total, 0 up, 0 in 2024-05-08T19:59:36.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:36 smithi083 ceph-mon[38695]: mgrmap e6: smithi083.fgcgxm(active, starting, since 0.00569524s) 2024-05-08T19:59:36.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:36 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi083"}]: dispatch 2024-05-08T19:59:36.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:36 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mgr metadata", "who": "smithi083.fgcgxm", "id": "smithi083.fgcgxm"}]: dispatch 2024-05-08T19:59:36.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:36 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-08T19:59:36.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:36 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-08T19:59:36.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:36 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-08T19:59:36.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:36 smithi083 ceph-mon[38695]: Manager daemon smithi083.fgcgxm is now available 2024-05-08T19:59:36.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:36 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' 2024-05-08T19:59:36.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:36 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' 2024-05-08T19:59:36.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:36 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T19:59:37.584 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:37 smithi083 ceph-mon[38695]: Found migration_current of "None". Setting to last migration. 2024-05-08T19:59:37.584 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:37 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T19:59:37.584 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:37 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.fgcgxm/mirror_snapshot_schedule"}]: dispatch 2024-05-08T19:59:37.585 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:37 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.fgcgxm/trash_purge_schedule"}]: dispatch 2024-05-08T19:59:37.585 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:37 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' 2024-05-08T19:59:37.585 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:37 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' 2024-05-08T19:59:37.585 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:37 smithi083 ceph-mon[38695]: mgrmap e7: smithi083.fgcgxm(active, since 1.00497s) 2024-05-08T19:59:37.821 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout { 2024-05-08T19:59:37.821 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-05-08T19:59:37.821 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-08T19:59:37.821 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout } 2024-05-08T19:59:37.821 INFO:teuthology.orchestra.run.smithi083.stdout:mgr epoch 5 is available 2024-05-08T19:59:37.821 INFO:teuthology.orchestra.run.smithi083.stdout:Setting orchestrator backend to cephadm... 2024-05-08T19:59:38.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:38 smithi083 ceph-mon[38695]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-08T19:59:38.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:38 smithi083 ceph-mon[38695]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-08T19:59:38.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:38 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T19:59:39.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:39 smithi083 ceph-mon[38695]: [08/May/2024:19:59:37] ENGINE Bus STARTING 2024-05-08T19:59:39.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:39 smithi083 ceph-mon[38695]: [08/May/2024:19:59:37] ENGINE Serving on http://172.21.15.83:8765 2024-05-08T19:59:39.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:39 smithi083 ceph-mon[38695]: [08/May/2024:19:59:37] ENGINE Serving on https://172.21.15.83:7150 2024-05-08T19:59:39.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:39 smithi083 ceph-mon[38695]: [08/May/2024:19:59:37] ENGINE Bus STARTED 2024-05-08T19:59:39.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:39 smithi083 ceph-mon[38695]: mgrmap e8: smithi083.fgcgxm(active, since 2s) 2024-05-08T19:59:39.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:39 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' 2024-05-08T19:59:39.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:39 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T19:59:40.401 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout value unchanged 2024-05-08T19:59:40.401 INFO:teuthology.orchestra.run.smithi083.stdout:Generating ssh key... 2024-05-08T19:59:40.403 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:40 smithi083 ceph-mon[38695]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T19:59:41.511 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:41 smithi083 ceph-mon[38695]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T19:59:42.328 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:42 smithi083 ceph-mon[38695]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T19:59:42.328 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:42 smithi083 ceph-mon[38695]: Generating ssh key... 2024-05-08T19:59:42.328 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:42 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' 2024-05-08T19:59:42.328 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:42 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' 2024-05-08T19:59:43.438 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKMfsUM72CbRMYwrVPPHwdQ+OCs9R0gjOzLHcYuHZInudr3a1KZAoNFsp7y51Ea8CBiPXXsog0gGEDWQWUy1pYXs2BTI6CBblB0cOGu3qa1GtcQhPhBFx+66z1v9c1hOIdvuDPzZ/NPOpGbdBSNMCaVfp9/Oi+RMDPT3CtwNvsLsJM9YQMH0fVtxFFBLZ3pRzK0f6ripbN+8BUdZ3Q5YUAKdNKi6X6uk+CjRhdgPAzy7usObZ22+AuIycIW2HjHCIVEBVW1pmf2hnNwpRw69c2VLCTVj7tf2SBxy1BTCoKqqsqCjTpoTtSAgrbvblx9M/Xl/cdQg5wgv92I5Qu7jzoXLa0c4QBuIN2YKy+gI4/lx+NIXU8c14YZnw4o92essyQD+gcHDo8V0cj8wppoPPbWhBq3hjtwX93dcQombPzupl5SqG5OopxTxX98raCFTVsxB0uPzJgLbX9Pgslzofqp/0Sw12HerwGYLDjykC8PpAq1C12yeneF/IahZiN7uk= ceph-526038a6-0d75-11ef-bc97-c7b262605968 2024-05-08T19:59:43.439 INFO:teuthology.orchestra.run.smithi083.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-08T19:59:43.439 INFO:teuthology.orchestra.run.smithi083.stdout:Adding key to root@localhost authorized_keys... 2024-05-08T19:59:43.440 INFO:teuthology.orchestra.run.smithi083.stdout:Adding host smithi083... 2024-05-08T19:59:43.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:43 smithi083 ceph-mon[38695]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T19:59:44.742 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:44 smithi083 ceph-mon[38695]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi083", "addr": "172.21.15.83", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T19:59:45.918 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:45 smithi083 ceph-mon[38695]: Deploying cephadm binary to smithi083 2024-05-08T19:59:46.980 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout Added host 'smithi083' with addr '172.21.15.83' 2024-05-08T19:59:46.980 INFO:teuthology.orchestra.run.smithi083.stdout:Deploying mon service with default placement... 2024-05-08T19:59:47.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:47 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' 2024-05-08T19:59:47.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:47 smithi083 ceph-mon[38695]: Added host smithi083 2024-05-08T19:59:47.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:47 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T19:59:49.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:48 smithi083 ceph-mon[38695]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T19:59:49.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:48 smithi083 ceph-mon[38695]: Saving service mon spec with placement count:5 2024-05-08T19:59:49.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:48 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' 2024-05-08T19:59:49.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:48 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' 2024-05-08T19:59:49.240 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-05-08T19:59:49.246 INFO:teuthology.orchestra.run.smithi083.stdout:Deploying mgr service with default placement... 2024-05-08T19:59:50.718 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-05-08T19:59:50.718 INFO:teuthology.orchestra.run.smithi083.stdout:Deploying crash service with default placement... 2024-05-08T19:59:50.720 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:50 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' 2024-05-08T19:59:50.721 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:50 smithi083 ceph-mon[38695]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T19:59:50.721 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:50 smithi083 ceph-mon[38695]: Saving service mgr spec with placement count:2 2024-05-08T19:59:50.721 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:50 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' 2024-05-08T19:59:52.218 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-05-08T19:59:52.219 INFO:teuthology.orchestra.run.smithi083.stdout:Deploying ceph-exporter service with default placement... 2024-05-08T19:59:52.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:52 smithi083 ceph-mon[38695]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T19:59:52.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:52 smithi083 ceph-mon[38695]: Saving service crash spec with placement * 2024-05-08T19:59:52.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:52 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' 2024-05-08T19:59:52.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:52 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' 2024-05-08T19:59:53.650 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-05-08T19:59:53.651 INFO:teuthology.orchestra.run.smithi083.stdout:Deploying prometheus service with default placement... 2024-05-08T19:59:54.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:53 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' 2024-05-08T19:59:54.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:53 smithi083 ceph-mon[38695]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T19:59:54.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:53 smithi083 ceph-mon[38695]: Saving service ceph-exporter spec with placement * 2024-05-08T19:59:54.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:53 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' 2024-05-08T19:59:56.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:55 smithi083 ceph-mon[38695]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T19:59:56.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:55 smithi083 ceph-mon[38695]: Saving service prometheus spec with placement count:1 2024-05-08T19:59:56.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:55 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' 2024-05-08T19:59:56.789 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-05-08T19:59:56.789 INFO:teuthology.orchestra.run.smithi083.stdout:Deploying grafana service with default placement... 2024-05-08T19:59:57.557 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:57 smithi083 ceph-mon[38695]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T19:59:58.079 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-05-08T19:59:58.079 INFO:teuthology.orchestra.run.smithi083.stdout:Deploying node-exporter service with default placement... 2024-05-08T19:59:58.868 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:58 smithi083 ceph-mon[38695]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T19:59:58.868 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:58 smithi083 ceph-mon[38695]: Saving service grafana spec with placement count:1 2024-05-08T19:59:58.868 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:58 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' 2024-05-08T19:59:58.868 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:58 smithi083 ceph-mon[38695]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T19:59:59.402 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-05-08T19:59:59.402 INFO:teuthology.orchestra.run.smithi083.stdout:Deploying alertmanager service with default placement... 2024-05-08T19:59:59.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:59 smithi083 ceph-mon[38695]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T19:59:59.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:59 smithi083 ceph-mon[38695]: Saving service node-exporter spec with placement * 2024-05-08T19:59:59.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 19:59:59 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:01.001 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-05-08T20:00:01.269 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:01 smithi083 ceph-mon[38695]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:00:01.269 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:01 smithi083 ceph-mon[38695]: Saving service alertmanager spec with placement count:1 2024-05-08T20:00:01.269 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:01 smithi083 ceph-mon[38695]: from='mgr.14118 172.21.15.83:0/1556818281' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:01.269 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:01 smithi083 ceph-mon[38695]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:00:02.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:02 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2112739338' entity='client.admin' 2024-05-08T20:00:02.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:02 smithi083 ceph-mon[38695]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:00:03.657 INFO:teuthology.orchestra.run.smithi083.stdout:Enabling the dashboard module... 2024-05-08T20:00:04.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:04 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/1860087026' entity='client.admin' 2024-05-08T20:00:05.420 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:05 smithi083 ceph-mon[38695]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:00:05.420 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:05 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2079338928' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-08T20:00:06.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:06 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2079338928' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-08T20:00:06.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:06 smithi083 ceph-mon[38695]: mgrmap e9: smithi083.fgcgxm(active, since 28s) 2024-05-08T20:00:07.163 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout { 2024-05-08T20:00:07.163 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-05-08T20:00:07.163 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-08T20:00:07.163 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "active_name": "smithi083.fgcgxm", 2024-05-08T20:00:07.163 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-08T20:00:07.163 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout } 2024-05-08T20:00:07.163 INFO:teuthology.orchestra.run.smithi083.stdout:Waiting for the mgr to restart... 2024-05-08T20:00:07.163 INFO:teuthology.orchestra.run.smithi083.stdout:Waiting for mgr epoch 9... 2024-05-08T20:00:07.164 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:07 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/4274854954' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-08T20:00:12.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:11 smithi083 ceph-mon[38695]: Active manager daemon smithi083.fgcgxm restarted 2024-05-08T20:00:12.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:11 smithi083 ceph-mon[38695]: Activating manager daemon smithi083.fgcgxm 2024-05-08T20:00:12.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:11 smithi083 ceph-mon[38695]: osdmap e3: 0 total, 0 up, 0 in 2024-05-08T20:00:12.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:11 smithi083 ceph-mon[38695]: mgrmap e10: smithi083.fgcgxm(active, starting, since 0.00562779s) 2024-05-08T20:00:12.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:11 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi083"}]: dispatch 2024-05-08T20:00:12.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:11 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mgr metadata", "who": "smithi083.fgcgxm", "id": "smithi083.fgcgxm"}]: dispatch 2024-05-08T20:00:12.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:11 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-08T20:00:12.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:11 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-08T20:00:12.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:11 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-08T20:00:12.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:11 smithi083 ceph-mon[38695]: Manager daemon smithi083.fgcgxm is now available 2024-05-08T20:00:12.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:11 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:00:13.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:12 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.fgcgxm/mirror_snapshot_schedule"}]: dispatch 2024-05-08T20:00:13.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:12 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.fgcgxm/trash_purge_schedule"}]: dispatch 2024-05-08T20:00:13.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:12 smithi083 ceph-mon[38695]: mgrmap e11: smithi083.fgcgxm(active, since 1.00533s) 2024-05-08T20:00:13.589 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout { 2024-05-08T20:00:13.589 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-05-08T20:00:13.590 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-08T20:00:13.590 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout } 2024-05-08T20:00:13.590 INFO:teuthology.orchestra.run.smithi083.stdout:mgr epoch 9 is available 2024-05-08T20:00:13.590 INFO:teuthology.orchestra.run.smithi083.stdout:Generating a dashboard self-signed certificate... 2024-05-08T20:00:14.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:13 smithi083 ceph-mon[38695]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-08T20:00:14.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:13 smithi083 ceph-mon[38695]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-08T20:00:14.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:13 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:15.139 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:15 smithi083 ceph-mon[38695]: [08/May/2024:20:00:13] ENGINE Bus STARTING 2024-05-08T20:00:15.139 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:15 smithi083 ceph-mon[38695]: [08/May/2024:20:00:14] ENGINE Serving on http://172.21.15.83:8765 2024-05-08T20:00:15.139 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:15 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:15.139 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:15 smithi083 ceph-mon[38695]: [08/May/2024:20:00:14] ENGINE Serving on https://172.21.15.83:7150 2024-05-08T20:00:15.139 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:15 smithi083 ceph-mon[38695]: [08/May/2024:20:00:14] ENGINE Bus STARTED 2024-05-08T20:00:15.140 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:15 smithi083 ceph-mon[38695]: mgrmap e12: smithi083.fgcgxm(active, since 2s) 2024-05-08T20:00:15.140 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:15 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:15.140 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:15 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:15.804 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-05-08T20:00:15.804 INFO:teuthology.orchestra.run.smithi083.stdout:Creating initial admin user... 2024-05-08T20:00:16.062 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:16 smithi083 ceph-mon[38695]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:00:17.469 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$sQcugFBMNCXe7wMwB8crD.2rQbShvOtHcM1AMQi8bM5TUR4zf/vzS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1715198416, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-08T20:00:17.469 INFO:teuthology.orchestra.run.smithi083.stdout:Fetching dashboard port number... 2024-05-08T20:00:18.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:17 smithi083 ceph-mon[38695]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:00:18.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:17 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:18.826 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 8443 2024-05-08T20:00:18.836 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-08T20:00:18.836 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout disabled 2024-05-08T20:00:18.844 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-08T20:00:18.844 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout inactive 2024-05-08T20:00:18.844 INFO:teuthology.orchestra.run.smithi083.stdout:firewalld.service is not enabled 2024-05-08T20:00:18.844 INFO:teuthology.orchestra.run.smithi083.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-08T20:00:18.849 INFO:teuthology.orchestra.run.smithi083.stdout:Ceph Dashboard is now available at: 2024-05-08T20:00:18.849 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:00:18.849 INFO:teuthology.orchestra.run.smithi083.stdout: URL: https://smithi083.front.sepia.ceph.com:8443/ 2024-05-08T20:00:18.849 INFO:teuthology.orchestra.run.smithi083.stdout: User: admin 2024-05-08T20:00:18.849 INFO:teuthology.orchestra.run.smithi083.stdout: Password: 9xbw3i64bx 2024-05-08T20:00:18.849 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:00:18.850 INFO:teuthology.orchestra.run.smithi083.stdout:Saving cluster configuration to /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/config directory 2024-05-08T20:00:18.850 INFO:teuthology.orchestra.run.smithi083.stdout:Enabling autotune for osd_memory_target 2024-05-08T20:00:19.100 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:18 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2394871864' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-08T20:00:19.904 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:19 smithi083 ceph-mon[38695]: mgrmap e13: smithi083.fgcgxm(active, since 6s) 2024-05-08T20:00:21.639 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-05-08T20:00:21.639 INFO:teuthology.orchestra.run.smithi083.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-05-08T20:00:21.639 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:00:21.639 INFO:teuthology.orchestra.run.smithi083.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-08T20:00:21.639 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:00:21.640 INFO:teuthology.orchestra.run.smithi083.stdout:Or, if you are only running a single cluster on this host: 2024-05-08T20:00:21.640 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:00:21.640 INFO:teuthology.orchestra.run.smithi083.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-05-08T20:00:21.640 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:00:21.640 INFO:teuthology.orchestra.run.smithi083.stdout:Please consider enabling telemetry to help improve Ceph: 2024-05-08T20:00:21.640 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:00:21.640 INFO:teuthology.orchestra.run.smithi083.stdout: ceph telemetry on 2024-05-08T20:00:21.640 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:00:21.640 INFO:teuthology.orchestra.run.smithi083.stdout:For more information see: 2024-05-08T20:00:21.640 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:00:21.640 INFO:teuthology.orchestra.run.smithi083.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-05-08T20:00:21.640 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:00:21.640 INFO:teuthology.orchestra.run.smithi083.stdout:Bootstrap complete. 2024-05-08T20:00:21.675 INFO:tasks.cephadm:Fetching config... 2024-05-08T20:00:21.675 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-05-08T20:00:21.675 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-08T20:00:21.693 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-08T20:00:21.693 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-05-08T20:00:21.693 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-08T20:00:21.751 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-08T20:00:21.751 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-05-08T20:00:21.751 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/keyring of=/dev/stdout 2024-05-08T20:00:21.819 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-08T20:00:21.819 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-05-08T20:00:21.819 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-08T20:00:21.877 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-08T20:00:21.877 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKMfsUM72CbRMYwrVPPHwdQ+OCs9R0gjOzLHcYuHZInudr3a1KZAoNFsp7y51Ea8CBiPXXsog0gGEDWQWUy1pYXs2BTI6CBblB0cOGu3qa1GtcQhPhBFx+66z1v9c1hOIdvuDPzZ/NPOpGbdBSNMCaVfp9/Oi+RMDPT3CtwNvsLsJM9YQMH0fVtxFFBLZ3pRzK0f6ripbN+8BUdZ3Q5YUAKdNKi6X6uk+CjRhdgPAzy7usObZ22+AuIycIW2HjHCIVEBVW1pmf2hnNwpRw69c2VLCTVj7tf2SBxy1BTCoKqqsqCjTpoTtSAgrbvblx9M/Xl/cdQg5wgv92I5Qu7jzoXLa0c4QBuIN2YKy+gI4/lx+NIXU8c14YZnw4o92essyQD+gcHDo8V0cj8wppoPPbWhBq3hjtwX93dcQombPzupl5SqG5OopxTxX98raCFTVsxB0uPzJgLbX9Pgslzofqp/0Sw12HerwGYLDjykC8PpAq1C12yeneF/IahZiN7uk= ceph-526038a6-0d75-11ef-bc97-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-08T20:00:21.953 INFO:teuthology.orchestra.run.smithi083.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKMfsUM72CbRMYwrVPPHwdQ+OCs9R0gjOzLHcYuHZInudr3a1KZAoNFsp7y51Ea8CBiPXXsog0gGEDWQWUy1pYXs2BTI6CBblB0cOGu3qa1GtcQhPhBFx+66z1v9c1hOIdvuDPzZ/NPOpGbdBSNMCaVfp9/Oi+RMDPT3CtwNvsLsJM9YQMH0fVtxFFBLZ3pRzK0f6ripbN+8BUdZ3Q5YUAKdNKi6X6uk+CjRhdgPAzy7usObZ22+AuIycIW2HjHCIVEBVW1pmf2hnNwpRw69c2VLCTVj7tf2SBxy1BTCoKqqsqCjTpoTtSAgrbvblx9M/Xl/cdQg5wgv92I5Qu7jzoXLa0c4QBuIN2YKy+gI4/lx+NIXU8c14YZnw4o92essyQD+gcHDo8V0cj8wppoPPbWhBq3hjtwX93dcQombPzupl5SqG5OopxTxX98raCFTVsxB0uPzJgLbX9Pgslzofqp/0Sw12HerwGYLDjykC8PpAq1C12yeneF/IahZiN7uk= ceph-526038a6-0d75-11ef-bc97-c7b262605968 2024-05-08T20:00:21.964 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKMfsUM72CbRMYwrVPPHwdQ+OCs9R0gjOzLHcYuHZInudr3a1KZAoNFsp7y51Ea8CBiPXXsog0gGEDWQWUy1pYXs2BTI6CBblB0cOGu3qa1GtcQhPhBFx+66z1v9c1hOIdvuDPzZ/NPOpGbdBSNMCaVfp9/Oi+RMDPT3CtwNvsLsJM9YQMH0fVtxFFBLZ3pRzK0f6ripbN+8BUdZ3Q5YUAKdNKi6X6uk+CjRhdgPAzy7usObZ22+AuIycIW2HjHCIVEBVW1pmf2hnNwpRw69c2VLCTVj7tf2SBxy1BTCoKqqsqCjTpoTtSAgrbvblx9M/Xl/cdQg5wgv92I5Qu7jzoXLa0c4QBuIN2YKy+gI4/lx+NIXU8c14YZnw4o92essyQD+gcHDo8V0cj8wppoPPbWhBq3hjtwX93dcQombPzupl5SqG5OopxTxX98raCFTVsxB0uPzJgLbX9Pgslzofqp/0Sw12HerwGYLDjykC8PpAq1C12yeneF/IahZiN7uk= ceph-526038a6-0d75-11ef-bc97-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-08T20:00:22.010 INFO:teuthology.orchestra.run.smithi188.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKMfsUM72CbRMYwrVPPHwdQ+OCs9R0gjOzLHcYuHZInudr3a1KZAoNFsp7y51Ea8CBiPXXsog0gGEDWQWUy1pYXs2BTI6CBblB0cOGu3qa1GtcQhPhBFx+66z1v9c1hOIdvuDPzZ/NPOpGbdBSNMCaVfp9/Oi+RMDPT3CtwNvsLsJM9YQMH0fVtxFFBLZ3pRzK0f6ripbN+8BUdZ3Q5YUAKdNKi6X6uk+CjRhdgPAzy7usObZ22+AuIycIW2HjHCIVEBVW1pmf2hnNwpRw69c2VLCTVj7tf2SBxy1BTCoKqqsqCjTpoTtSAgrbvblx9M/Xl/cdQg5wgv92I5Qu7jzoXLa0c4QBuIN2YKy+gI4/lx+NIXU8c14YZnw4o92essyQD+gcHDo8V0cj8wppoPPbWhBq3hjtwX93dcQombPzupl5SqG5OopxTxX98raCFTVsxB0uPzJgLbX9Pgslzofqp/0Sw12HerwGYLDjykC8PpAq1C12yeneF/IahZiN7uk= ceph-526038a6-0d75-11ef-bc97-c7b262605968 2024-05-08T20:00:22.021 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-08T20:00:22.298 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:00:22.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:21 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/801088948' entity='client.admin' 2024-05-08T20:00:25.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:24 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:25.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:24 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:25.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:24 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd/host:smithi083", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:00:25.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:24 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:25.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:24 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi083", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-05-08T20:00:25.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:24 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi083", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-05-08T20:00:25.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:24 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:00:25.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:24 smithi083 ceph-mon[38695]: Deploying daemon ceph-exporter.smithi083 on smithi083 2024-05-08T20:00:26.553 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-08T20:00:26.553 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-08T20:00:26.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:26 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/4275640577' entity='client.admin' 2024-05-08T20:00:27.359 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:00:30.036 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:29 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:30.036 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:29 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:30.037 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:29 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:30.037 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:29 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:30.037 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:29 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi083", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-05-08T20:00:30.037 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:29 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi083", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-05-08T20:00:30.037 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:29 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:00:30.037 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:29 smithi083 ceph-mon[38695]: Deploying daemon crash.smithi083 on smithi083 2024-05-08T20:00:31.537 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi188 2024-05-08T20:00:31.537 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-05-08T20:00:31.537 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/etc/ceph/ceph.conf 2024-05-08T20:00:31.559 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-05-08T20:00:31.559 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-08T20:00:31.618 INFO:tasks.cephadm:Adding host smithi188 to orchestrator... 2024-05-08T20:00:31.619 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph orch host add smithi188 2024-05-08T20:00:31.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:31 smithi083 ceph-mon[38695]: from='client.14187 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:00:31.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:31 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:31.873 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:00:33.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:32 smithi083 ceph-mon[38695]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:00:33.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:32 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:34.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:34 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:34.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:34 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:34.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:34 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:34.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:34 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:34.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:34 smithi083 ceph-mon[38695]: Deploying daemon node-exporter.smithi083 on smithi083 2024-05-08T20:00:35.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:35 smithi083 ceph-mon[38695]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:00:36.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:36 smithi083 ceph-mon[38695]: from='client.14189 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:00:36.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:36 smithi083 ceph-mon[38695]: Deploying cephadm binary to smithi188 2024-05-08T20:00:36.497 INFO:teuthology.orchestra.run.smithi083.stdout:Added host 'smithi188' with addr '172.21.15.188' 2024-05-08T20:00:37.114 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph orch host ls --format=json 2024-05-08T20:00:37.405 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:37 smithi083 ceph-mon[38695]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:00:37.405 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:37 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:37.405 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:37 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:37.700 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:00:38.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:38 smithi083 ceph-mon[38695]: Added host smithi188 2024-05-08T20:00:38.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:38 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:38.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:38 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:38.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:38 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:38.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:38 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:39.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:39 smithi083 ceph-mon[38695]: Deploying daemon alertmanager.smithi083 on smithi083 2024-05-08T20:00:39.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:39 smithi083 ceph-mon[38695]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:00:39.753 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:00:39.753 INFO:teuthology.orchestra.run.smithi083.stdout:[{"addr": "172.21.15.83", "hostname": "smithi083", "labels": [], "status": ""}, {"addr": "172.21.15.188", "hostname": "smithi188", "labels": [], "status": ""}] 2024-05-08T20:00:41.014 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-08T20:00:41.014 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph osd crush tunables default 2024-05-08T20:00:41.233 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:00:41.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:41 smithi083 ceph-mon[38695]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-08T20:00:41.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:41 smithi083 ceph-mon[38695]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:00:43.305 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:42 smithi083 ceph-mon[38695]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:00:43.305 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:42 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:45.112 INFO:teuthology.orchestra.run.smithi083.stderr:adjusted tunables profile to default 2024-05-08T20:00:45.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:44 smithi083 ceph-mon[38695]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:00:45.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:44 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3539902638' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-08T20:00:46.176 INFO:tasks.cephadm:Adding mon.smithi083 on smithi083 2024-05-08T20:00:46.177 INFO:tasks.cephadm:Adding mon.smithi188 on smithi188 2024-05-08T20:00:46.177 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph orch apply mon '2;smithi083:172.21.15.83=smithi083;smithi188:172.21.15.188=smithi188' 2024-05-08T20:00:46.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:45 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3539902638' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-08T20:00:46.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:45 smithi083 ceph-mon[38695]: osdmap e4: 0 total, 0 up, 0 in 2024-05-08T20:00:46.430 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:00:46.492 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:00:47.294 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:46 smithi083 ceph-mon[38695]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:00:48.406 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mon update... 2024-05-08T20:00:49.014 DEBUG:teuthology.orchestra.run.smithi188:mon.smithi188> sudo journalctl -f -n 0 -u ceph-526038a6-0d75-11ef-bc97-c7b262605968@mon.smithi188.service 2024-05-08T20:00:49.016 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-08T20:00:49.016 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-08T20:00:49.046 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:-- Logs begin at Wed 2024-05-08 19:48:17 UTC. -- 2024-05-08T20:00:49.268 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:00:49.326 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:00:49.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:49 smithi083 ceph-mon[38695]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:00:49.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:49 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:49.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:49 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:49.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:49 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:49.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:49 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:49.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:49 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:49.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:49 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:49.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:49 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:49.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:49 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:49.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:49 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-08T20:00:49.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:49 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:49.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:49 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:50.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:50 smithi083 ceph-mon[38695]: Regenerating cephadm self-signed grafana TLS certificates 2024-05-08T20:00:50.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:50 smithi083 ceph-mon[38695]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-08T20:00:50.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:50 smithi083 ceph-mon[38695]: Deploying daemon grafana.smithi083 on smithi083 2024-05-08T20:00:50.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:50 smithi083 ceph-mon[38695]: from='client.14195 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi083:172.21.15.83=smithi083;smithi188:172.21.15.188=smithi188", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:00:50.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:50 smithi083 ceph-mon[38695]: Saving service mon spec with placement smithi083:172.21.15.83=smithi083;smithi188:172.21.15.188=smithi188;count:2 2024-05-08T20:00:51.394 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T20:00:51.394 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","modified":"2024-05-08T19:59:04.238534Z","created":"2024-05-08T19:59:04.238534Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-08T20:00:51.396 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-05-08T20:00:51.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:51 smithi083 ceph-mon[38695]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:00:52.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:52 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/4153930278' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-08T20:00:52.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:52 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:00:52.954 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-08T20:00:52.955 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-08T20:00:53.201 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:00:53.266 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:00:53.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:53 smithi083 ceph-mon[38695]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:00:55.142 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T20:00:55.142 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","modified":"2024-05-08T19:59:04.238534Z","created":"2024-05-08T19:59:04.238534Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-08T20:00:55.143 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-05-08T20:00:55.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:55 smithi083 ceph-mon[38695]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:00:56.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:56 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/273645240' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-08T20:00:56.736 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-08T20:00:56.736 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-08T20:00:56.981 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:00:57.042 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:00:57.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:57 smithi083 ceph-mon[38695]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:00:58.846 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T20:00:58.846 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","modified":"2024-05-08T19:59:04.238534Z","created":"2024-05-08T19:59:04.238534Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-08T20:00:58.848 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-05-08T20:00:59.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:59 smithi083 ceph-mon[38695]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:00:59.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:00:59 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/4075123520' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-08T20:01:00.485 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-08T20:01:00.486 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-08T20:01:00.734 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:01:00.799 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:01:01.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:01 smithi083 ceph-mon[38695]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:02.757 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T20:01:02.757 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","modified":"2024-05-08T19:59:04.238534Z","created":"2024-05-08T19:59:04.238534Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-08T20:01:02.759 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-05-08T20:01:03.347 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:03 smithi083 ceph-mon[38695]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:03.347 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:03 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/1560588814' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-08T20:01:04.382 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-08T20:01:04.382 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-08T20:01:04.632 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:01:04.695 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:01:04.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:04 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:04.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:04 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:04.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:04 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:04.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:04 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:04.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:04 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:04.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:04 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:04.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:04 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:04.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:04 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:04.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:04 smithi083 ceph-mon[38695]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:04.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:04 smithi083 ceph-mon[38695]: Deploying daemon prometheus.smithi083 on smithi083 2024-05-08T20:01:06.691 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T20:01:06.691 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","modified":"2024-05-08T19:59:04.238534Z","created":"2024-05-08T19:59:04.238534Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-08T20:01:06.693 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-05-08T20:01:07.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:06 smithi083 ceph-mon[38695]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:07.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:06 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/2727295389' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-08T20:01:08.354 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-08T20:01:08.354 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-08T20:01:08.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:07 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:08.617 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:01:08.678 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:01:09.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:08 smithi083 ceph-mon[38695]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:10.619 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T20:01:10.619 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","modified":"2024-05-08T19:59:04.238534Z","created":"2024-05-08T19:59:04.238534Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-08T20:01:10.622 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-05-08T20:01:11.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:10 smithi083 ceph-mon[38695]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:11.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:10 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/1464236928' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-08T20:01:12.290 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-08T20:01:12.291 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-08T20:01:12.546 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:01:12.611 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:01:13.428 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:12 smithi083 ceph-mon[38695]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:14.567 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T20:01:14.567 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","modified":"2024-05-08T19:59:04.238534Z","created":"2024-05-08T19:59:04.238534Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-08T20:01:14.569 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-05-08T20:01:15.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:14 smithi083 ceph-mon[38695]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:15.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:14 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/2322660636' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-08T20:01:16.202 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-08T20:01:16.203 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-08T20:01:16.452 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:01:16.512 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:01:17.324 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:16 smithi083 ceph-mon[38695]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:17.324 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:16 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:17.324 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:16 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:17.325 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:16 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:17.325 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:16 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-05-08T20:01:18.370 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:17 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:18.370 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:17 smithi083 ceph-mon[38695]: from='mgr.14162 172.21.15.83:0/2905278358' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-05-08T20:01:18.370 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:17 smithi083 ceph-mon[38695]: mgrmap e14: smithi083.fgcgxm(active, since 65s) 2024-05-08T20:01:18.691 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T20:01:18.692 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","modified":"2024-05-08T19:59:04.238534Z","created":"2024-05-08T19:59:04.238534Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-08T20:01:18.693 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-05-08T20:01:19.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:18 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/2488318043' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-08T20:01:20.243 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-08T20:01:20.243 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-08T20:01:20.502 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:01:20.561 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:01:22.369 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T20:01:22.370 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","modified":"2024-05-08T19:59:04.238534Z","created":"2024-05-08T19:59:04.238534Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-08T20:01:22.370 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-05-08T20:01:22.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:22 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/825391047' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-08T20:01:23.902 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-08T20:01:23.902 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-08T20:01:23.905 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:23 smithi083 ceph-mon[38695]: Active manager daemon smithi083.fgcgxm restarted 2024-05-08T20:01:23.905 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:23 smithi083 ceph-mon[38695]: Activating manager daemon smithi083.fgcgxm 2024-05-08T20:01:23.905 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:23 smithi083 ceph-mon[38695]: osdmap e5: 0 total, 0 up, 0 in 2024-05-08T20:01:23.905 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:23 smithi083 ceph-mon[38695]: mgrmap e15: smithi083.fgcgxm(active, starting, since 0.00558339s) 2024-05-08T20:01:23.905 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi083"}]: dispatch 2024-05-08T20:01:23.905 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mgr metadata", "who": "smithi083.fgcgxm", "id": "smithi083.fgcgxm"}]: dispatch 2024-05-08T20:01:23.905 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-08T20:01:23.905 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-08T20:01:23.905 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-08T20:01:23.905 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:23 smithi083 ceph-mon[38695]: Manager daemon smithi083.fgcgxm is now available 2024-05-08T20:01:23.905 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:01:24.153 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:01:24.216 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:01:25.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:25.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:01:25.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.fgcgxm/mirror_snapshot_schedule"}]: dispatch 2024-05-08T20:01:25.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.fgcgxm/trash_purge_schedule"}]: dispatch 2024-05-08T20:01:25.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:25.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:24 smithi083 ceph-mon[38695]: mgrmap e16: smithi083.fgcgxm(active, since 1.00631s) 2024-05-08T20:01:26.192 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T20:01:26.192 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","modified":"2024-05-08T19:59:04.238534Z","created":"2024-05-08T19:59:04.238534Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-08T20:01:26.195 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-05-08T20:01:26.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:26 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:26.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:26 smithi083 ceph-mon[38695]: [08/May/2024:20:01:25] ENGINE Bus STARTING 2024-05-08T20:01:26.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:26 smithi083 ceph-mon[38695]: [08/May/2024:20:01:25] ENGINE Serving on https://172.21.15.83:7150 2024-05-08T20:01:26.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:26 smithi083 ceph-mon[38695]: [08/May/2024:20:01:26] ENGINE Serving on http://172.21.15.83:8765 2024-05-08T20:01:26.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:26 smithi083 ceph-mon[38695]: [08/May/2024:20:01:26] ENGINE Bus STARTED 2024-05-08T20:01:26.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:26 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:26.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:26 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/2857143498' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-08T20:01:27.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:27 smithi083 ceph-mon[38695]: mgrmap e17: smithi083.fgcgxm(active, since 2s) 2024-05-08T20:01:27.833 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-08T20:01:27.833 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-08T20:01:28.307 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:01:28.363 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-08T20:01:29.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:29 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:29.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:29 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:29.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:29 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:29.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:29 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:29.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:29 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd/host:smithi083", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:01:30.566 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T20:01:30.566 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","modified":"2024-05-08T19:59:04.238534Z","created":"2024-05-08T19:59:04.238534Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-08T20:01:30.569 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-05-08T20:01:30.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:30 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/2750887816' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-08T20:01:30.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:30 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:30.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:30 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:30.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:30 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:30.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:30 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:30.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:30 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd/host:smithi188", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:01:30.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:30 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:01:30.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:30 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:01:31.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:31 smithi083 ceph-mon[38695]: Updating smithi083:/etc/ceph/ceph.conf 2024-05-08T20:01:31.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:31 smithi083 ceph-mon[38695]: Updating smithi188:/etc/ceph/ceph.conf 2024-05-08T20:01:31.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:31 smithi083 ceph-mon[38695]: Updating smithi188:/var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-08T20:01:31.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:31 smithi083 ceph-mon[38695]: Updating smithi083:/var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-08T20:01:31.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:31 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:31.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:31 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:32.527 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-08T20:01:32.528 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-08T20:01:32.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:32 smithi083 ceph-mon[38695]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-05-08T20:01:32.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:32 smithi083 ceph-mon[38695]: Updating smithi188:/var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-08T20:01:32.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:32 smithi083 ceph-mon[38695]: Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-05-08T20:01:32.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:32 smithi083 ceph-mon[38695]: Updating smithi083:/var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-08T20:01:32.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:32 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:32.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:32 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:32.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:32 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:32.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:32 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi188", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-05-08T20:01:32.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:32 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi188", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-05-08T20:01:32.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:32 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:01:33.590 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-08T20:01:33.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:33 smithi083 ceph-mon[38695]: Deploying daemon ceph-exporter.smithi188 on smithi188 2024-05-08T20:01:35.640 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T20:01:35.641 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","modified":"2024-05-08T19:59:04.238534Z","created":"2024-05-08T19:59:04.238534Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-08T20:01:35.642 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-05-08T20:01:36.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:35 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/3792968501' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-08T20:01:37.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:37 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:37.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:37 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:37.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:37 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:37.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:37 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:37.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:37 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi188", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-05-08T20:01:37.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:37 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi188", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-05-08T20:01:37.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:37 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:01:37.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:37 smithi083 ceph-mon[38695]: Deploying daemon crash.smithi188 on smithi188 2024-05-08T20:01:37.442 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-08T20:01:37.443 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-08T20:01:37.679 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-08T20:01:40.008 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T20:01:40.008 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","modified":"2024-05-08T19:59:04.238534Z","created":"2024-05-08T19:59:04.238534Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-08T20:01:40.018 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-05-08T20:01:40.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:40.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:01:40.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:40.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:40.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:40.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:41.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:40 smithi083 ceph-mon[38695]: Deploying daemon node-exporter.smithi188 on smithi188 2024-05-08T20:01:41.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:40 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/2806103734' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-08T20:01:41.566 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-08T20:01:41.566 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-08T20:01:41.784 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-08T20:01:45.049 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T20:01:45.049 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","modified":"2024-05-08T19:59:04.238534Z","created":"2024-05-08T19:59:04.238534Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-08T20:01:45.051 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-05-08T20:01:45.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:44 smithi083 ceph-mon[38695]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:45.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:44 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:45.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:44 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:45.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:44 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:45.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:44 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:45.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:44 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:45.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:44 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi188.abfynx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-08T20:01:45.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:44 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi188.abfynx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-08T20:01:45.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:44 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-08T20:01:45.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:44 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:01:46.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:45 smithi083 ceph-mon[38695]: Deploying daemon mgr.smithi188.abfynx on smithi188 2024-05-08T20:01:46.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:45 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/841990849' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-08T20:01:47.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:46 smithi083 ceph-mon[38695]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:47.382 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-08T20:01:47.383 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-08T20:01:48.021 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-08T20:01:48.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:47 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:48.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:47 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:48.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:47 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:48.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:47 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:48.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:47 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-08T20:01:48.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:47 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:01:48.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:47 smithi083 ceph-mon[38695]: Deploying daemon mon.smithi188 on smithi188 2024-05-08T20:01:49.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:48 smithi083 ceph-mon[38695]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:49.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:48 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:51.318 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 systemd[1]: Starting Ceph mon.smithi188 for 526038a6-0d75-11ef-bc97-c7b262605968... 2024-05-08T20:01:51.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:50 smithi083 ceph-mon[38695]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:51.896 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T20:01:51.896 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","modified":"2024-05-08T19:59:04.238534Z","created":"2024-05-08T19:59:04.238534Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-08T20:01:51.898 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-05-08T20:01:52.247 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: set uid:gid to 167:167 (ceph:ceph) 2024-05-08T20:01:52.248 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: ceph version 18.2.2-1322-gd21e4f96 (d21e4f961318b56f0d915caa2490c3711bf13c34) reef (stable), process ceph-mon, pid 7 2024-05-08T20:01:52.248 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: pidfile_write: ignore empty --pid-file 2024-05-08T20:01:52.248 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: load: jerasure load: lrc 2024-05-08T20:01:52.248 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: RocksDB version: 7.9.2 2024-05-08T20:01:52.248 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Git sha 0 2024-05-08T20:01:52.248 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Compile date 2024-05-08 17:55:50 2024-05-08T20:01:52.248 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: DB SUMMARY 2024-05-08T20:01:52.249 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: DB Session ID: S33XSAP5CDOV1A5CN334 2024-05-08T20:01:52.249 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: CURRENT file: CURRENT 2024-05-08T20:01:52.249 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: IDENTITY file: IDENTITY 2024-05-08T20:01:52.249 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-05-08T20:01:52.249 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi188/store.db dir, Total Num: 0, files: 2024-05-08T20:01:52.250 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi188/store.db: 000004.log size: 511 ; 2024-05-08T20:01:52.250 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.error_if_exists: 0 2024-05-08T20:01:52.250 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.create_if_missing: 0 2024-05-08T20:01:52.250 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.paranoid_checks: 1 2024-05-08T20:01:52.250 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-08T20:01:52.251 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-08T20:01:52.251 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-08T20:01:52.251 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.env: 0x5587c8903820 2024-05-08T20:01:52.251 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.fs: PosixFileSystem 2024-05-08T20:01:52.251 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.info_log: 0x5587ca2373e0 2024-05-08T20:01:52.251 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_file_opening_threads: 16 2024-05-08T20:01:52.251 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.statistics: (nil) 2024-05-08T20:01:52.252 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.use_fsync: 0 2024-05-08T20:01:52.252 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_log_file_size: 0 2024-05-08T20:01:52.252 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-08T20:01:52.252 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-08T20:01:52.252 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.keep_log_file_num: 1000 2024-05-08T20:01:52.252 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.recycle_log_file_num: 0 2024-05-08T20:01:52.252 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.allow_fallocate: 1 2024-05-08T20:01:52.253 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.allow_mmap_reads: 0 2024-05-08T20:01:52.253 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.allow_mmap_writes: 0 2024-05-08T20:01:52.253 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.use_direct_reads: 0 2024-05-08T20:01:52.253 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-08T20:01:52.253 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.create_missing_column_families: 0 2024-05-08T20:01:52.253 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.db_log_dir: 2024-05-08T20:01:52.253 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.wal_dir: 2024-05-08T20:01:52.253 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-08T20:01:52.253 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-08T20:01:52.254 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-08T20:01:52.254 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-08T20:01:52.254 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-08T20:01:52.254 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-08T20:01:52.254 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.advise_random_on_open: 1 2024-05-08T20:01:52.255 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.db_write_buffer_size: 0 2024-05-08T20:01:52.255 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.write_buffer_manager: 0x5587c94ac140 2024-05-08T20:01:52.255 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-08T20:01:52.255 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-08T20:01:52.255 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-08T20:01:52.255 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.rate_limiter: (nil) 2024-05-08T20:01:52.256 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-08T20:01:52.256 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.wal_recovery_mode: 2 2024-05-08T20:01:52.256 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.enable_thread_tracking: 0 2024-05-08T20:01:52.256 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.enable_pipelined_write: 0 2024-05-08T20:01:52.256 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.unordered_write: 0 2024-05-08T20:01:52.256 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-08T20:01:52.256 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-08T20:01:52.256 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-08T20:01:52.257 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-08T20:01:52.257 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.row_cache: None 2024-05-08T20:01:52.257 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.wal_filter: None 2024-05-08T20:01:52.257 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-08T20:01:52.257 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.allow_ingest_behind: 0 2024-05-08T20:01:52.257 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.two_write_queues: 0 2024-05-08T20:01:52.257 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.manual_wal_flush: 0 2024-05-08T20:01:52.258 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.wal_compression: 0 2024-05-08T20:01:52.258 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.atomic_flush: 0 2024-05-08T20:01:52.258 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-08T20:01:52.258 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-08T20:01:52.258 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-08T20:01:52.258 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.log_readahead_size: 0 2024-05-08T20:01:52.258 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-08T20:01:52.258 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.best_efforts_recovery: 0 2024-05-08T20:01:52.259 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-08T20:01:52.259 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-08T20:01:52.259 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.allow_data_in_errors: 0 2024-05-08T20:01:52.259 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.db_host_id: __hostname__ 2024-05-08T20:01:52.259 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-08T20:01:52.259 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_background_jobs: 2 2024-05-08T20:01:52.260 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_background_compactions: -1 2024-05-08T20:01:52.260 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_subcompactions: 1 2024-05-08T20:01:52.260 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-08T20:01:52.260 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-08T20:01:52.260 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-08T20:01:52.260 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_total_wal_size: 0 2024-05-08T20:01:52.260 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-08T20:01:52.260 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-08T20:01:52.260 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-08T20:01:52.260 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-08T20:01:52.260 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_open_files: -1 2024-05-08T20:01:52.261 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.bytes_per_sync: 0 2024-05-08T20:01:52.261 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-08T20:01:52.261 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-08T20:01:52.261 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.compaction_readahead_size: 0 2024-05-08T20:01:52.261 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_background_flushes: -1 2024-05-08T20:01:52.261 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Compression algorithms supported: 2024-05-08T20:01:52.261 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-08T20:01:52.261 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: kZSTD supported: 0 2024-05-08T20:01:52.261 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: kXpressCompression supported: 0 2024-05-08T20:01:52.261 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: kLZ4HCCompression supported: 1 2024-05-08T20:01:52.261 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: kZlibCompression supported: 1 2024-05-08T20:01:52.261 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: kSnappyCompression supported: 1 2024-05-08T20:01:52.261 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: kLZ4Compression supported: 1 2024-05-08T20:01:52.261 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: kBZip2Compression supported: 0 2024-05-08T20:01:52.262 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-08T20:01:52.262 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-08T20:01:52.262 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi188/store.db/MANIFEST-000005 2024-05-08T20:01:52.262 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-08T20:01:52.262 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-08T20:01:52.262 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.merge_operator: 2024-05-08T20:01:52.262 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.compaction_filter: None 2024-05-08T20:01:52.262 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.compaction_filter_factory: None 2024-05-08T20:01:52.262 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.sst_partitioner_factory: None 2024-05-08T20:01:52.262 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-08T20:01:52.262 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-08T20:01:52.262 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5587ca237500) 2024-05-08T20:01:52.262 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: cache_index_and_filter_blocks: 1 2024-05-08T20:01:52.263 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-08T20:01:52.263 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-08T20:01:52.263 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: pin_top_level_index_and_filter: 1 2024-05-08T20:01:52.263 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: index_type: 0 2024-05-08T20:01:52.263 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: data_block_index_type: 0 2024-05-08T20:01:52.263 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: index_shortening: 1 2024-05-08T20:01:52.263 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-08T20:01:52.263 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: checksum: 4 2024-05-08T20:01:52.263 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: no_block_cache: 0 2024-05-08T20:01:52.263 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: block_cache: 0x5587c954f1f0 2024-05-08T20:01:52.263 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: block_cache_name: BinnedLRUCache 2024-05-08T20:01:52.263 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: block_cache_options: 2024-05-08T20:01:52.263 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: capacity : 536870912 2024-05-08T20:01:52.263 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: num_shard_bits : 4 2024-05-08T20:01:52.264 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: strict_capacity_limit : 0 2024-05-08T20:01:52.264 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: high_pri_pool_ratio: 0.000 2024-05-08T20:01:52.264 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: block_cache_compressed: (nil) 2024-05-08T20:01:52.264 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: persistent_cache: (nil) 2024-05-08T20:01:52.264 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: block_size: 4096 2024-05-08T20:01:52.264 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: block_size_deviation: 10 2024-05-08T20:01:52.264 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: block_restart_interval: 16 2024-05-08T20:01:52.264 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: index_block_restart_interval: 1 2024-05-08T20:01:52.264 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: metadata_block_size: 4096 2024-05-08T20:01:52.264 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: partition_filters: 0 2024-05-08T20:01:52.264 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: use_delta_encoding: 1 2024-05-08T20:01:52.264 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: filter_policy: bloomfilter 2024-05-08T20:01:52.265 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: whole_key_filtering: 1 2024-05-08T20:01:52.265 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: verify_compression: 0 2024-05-08T20:01:52.265 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: read_amp_bytes_per_bit: 0 2024-05-08T20:01:52.265 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: format_version: 5 2024-05-08T20:01:52.265 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: enable_index_compression: 1 2024-05-08T20:01:52.265 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: block_align: 0 2024-05-08T20:01:52.265 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: max_auto_readahead_size: 262144 2024-05-08T20:01:52.265 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: prepopulate_block_cache: 0 2024-05-08T20:01:52.265 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: initial_auto_readahead_size: 8192 2024-05-08T20:01:52.265 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: num_file_reads_for_auto_readahead: 2 2024-05-08T20:01:52.265 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.write_buffer_size: 33554432 2024-05-08T20:01:52.265 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_write_buffer_number: 2 2024-05-08T20:01:52.265 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.compression: NoCompression 2024-05-08T20:01:52.265 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.bottommost_compression: Disabled 2024-05-08T20:01:52.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.prefix_extractor: nullptr 2024-05-08T20:01:52.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-08T20:01:52.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.num_levels: 7 2024-05-08T20:01:52.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-08T20:01:52.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-08T20:01:52.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-08T20:01:52.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-08T20:01:52.267 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-08T20:01:52.267 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-08T20:01:52.267 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-08T20:01:52.267 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-08T20:01:52.267 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-08T20:01:52.267 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-08T20:01:52.267 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-08T20:01:52.267 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-08T20:01:52.267 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-08T20:01:52.267 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.compression_opts.level: 32767 2024-05-08T20:01:52.267 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.compression_opts.strategy: 0 2024-05-08T20:01:52.267 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-08T20:01:52.267 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-08T20:01:52.267 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-08T20:01:52.268 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-08T20:01:52.268 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.compression_opts.enabled: false 2024-05-08T20:01:52.268 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-08T20:01:52.270 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-08T20:01:52.270 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-08T20:01:52.270 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-08T20:01:52.270 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.target_file_size_base: 67108864 2024-05-08T20:01:52.271 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-08T20:01:52.271 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-08T20:01:52.271 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-08T20:01:52.271 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-08T20:01:52.271 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-08T20:01:52.271 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-08T20:01:52.271 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-08T20:01:52.271 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-08T20:01:52.271 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-08T20:01:52.271 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-08T20:01:52.271 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-08T20:01:52.272 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-08T20:01:52.272 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-08T20:01:52.272 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-08T20:01:52.272 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.arena_block_size: 1048576 2024-05-08T20:01:52.272 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-08T20:01:52.272 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-08T20:01:52.272 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.disable_auto_compactions: 0 2024-05-08T20:01:52.272 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-08T20:01:52.272 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-08T20:01:52.272 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-08T20:01:52.272 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-08T20:01:52.272 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-08T20:01:52.272 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-08T20:01:52.272 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-08T20:01:52.273 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-08T20:01:52.273 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-08T20:01:52.273 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-08T20:01:52.273 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.table_properties_collectors: 2024-05-08T20:01:52.273 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.inplace_update_support: 0 2024-05-08T20:01:52.273 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-08T20:01:52.273 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-08T20:01:52.273 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-08T20:01:52.273 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-08T20:01:52.273 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.bloom_locality: 0 2024-05-08T20:01:52.273 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.max_successive_merges: 0 2024-05-08T20:01:52.273 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-08T20:01:52.273 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.paranoid_file_checks: 0 2024-05-08T20:01:52.273 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.force_consistency_checks: 1 2024-05-08T20:01:52.273 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.report_bg_io_stats: 0 2024-05-08T20:01:52.273 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.ttl: 2592000 2024-05-08T20:01:52.274 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-08T20:01:52.274 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-08T20:01:52.274 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-08T20:01:52.274 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.enable_blob_files: false 2024-05-08T20:01:52.274 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.min_blob_size: 0 2024-05-08T20:01:52.274 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.blob_file_size: 268435456 2024-05-08T20:01:52.274 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-08T20:01:52.274 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-08T20:01:52.274 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-08T20:01:52.274 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-08T20:01:52.275 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-08T20:01:52.275 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.blob_file_starting_level: 0 2024-05-08T20:01:52.275 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-08T20:01:52.275 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi188/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-08T20:01:52.275 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-05-08T20:01:52.275 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 57e0a57c-914e-4ef2-bdf5-b4d4a1a50754 2024-05-08T20:01:52.275 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715198511977040, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-08T20:01:52.275 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-05-08T20:01:52.275 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715198511977465, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715198511, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "57e0a57c-914e-4ef2-bdf5-b4d4a1a50754", "db_session_id": "S33XSAP5CDOV1A5CN334", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-05-08T20:01:52.275 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715198511977542, "job": 1, "event": "recovery_finished"} 2024-05-08T20:01:52.275 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-05-08T20:01:52.275 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi188/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-08T20:01:52.276 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5587c95ec000 2024-05-08T20:01:52.276 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: DB pointer 0x5587c95da000 2024-05-08T20:01:52.276 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-08T20:01:52.276 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-08T20:01:52.276 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: ** DB Stats ** 2024-05-08T20:01:52.276 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-08T20:01:52.276 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-08T20:01:52.276 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-08T20:01:52.276 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-08T20:01:52.276 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-08T20:01:52.276 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-08T20:01:52.276 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-08T20:01:52.277 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: 2024-05-08T20:01:52.277 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: ** Compaction Stats [default] ** 2024-05-08T20:01:52.277 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-08T20:01:52.277 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-08T20:01:52.277 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-08T20:01:52.277 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-08T20:01:52.277 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-08T20:01:52.277 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: 2024-05-08T20:01:52.277 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: ** Compaction Stats [default] ** 2024-05-08T20:01:52.277 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-08T20:01:52.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-08T20:01:52.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-08T20:01:52.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: 2024-05-08T20:01:52.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-08T20:01:52.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: 2024-05-08T20:01:52.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-08T20:01:52.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-08T20:01:52.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-08T20:01:52.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-08T20:01:52.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-08T20:01:52.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-08T20:01:52.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-08T20:01:52.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Interval compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-08T20:01:52.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-08T20:01:52.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Block cache BinnedLRUCache@0x5587c954f1f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-05-08T20:01:52.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-05-08T20:01:52.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: 2024-05-08T20:01:52.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-08T20:01:52.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: mon.smithi188 does not exist in monmap, will attempt to join an existing cluster 2024-05-08T20:01:52.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: using public_addr v2:172.21.15.188:0/0 -> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] 2024-05-08T20:01:52.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: starting mon.smithi188 rank -1 at public addrs [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] at bind addrs [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi188 fsid 526038a6-0d75-11ef-bc97-c7b262605968 2024-05-08T20:01:52.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:51 smithi188 ceph-mon[48761]: mon.smithi188@-1(???) e0 preinit fsid 526038a6-0d75-11ef-bc97-c7b262605968 2024-05-08T20:01:52.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: mon.smithi188@-1(synchronizing).mds e1 new map 2024-05-08T20:01:52.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: mon.smithi188@-1(synchronizing).mds e1 print_map 2024-05-08T20:01:52.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: e1 2024-05-08T20:01:52.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-08T20:01:52.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-08T20:01:52.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: legacy client fscid: -1 2024-05-08T20:01:52.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: 2024-05-08T20:01:52.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: No filesystems configured 2024-05-08T20:01:52.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: mon.smithi188@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-08T20:01:52.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: mon.smithi188@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-08T20:01:52.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: mon.smithi188@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-08T20:01:52.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: mon.smithi188@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-08T20:01:52.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: mon.smithi188@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-08T20:01:52.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: mon.smithi188@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-08T20:01:52.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: mon.smithi188@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-05-08T20:01:52.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: mon.smithi188@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-08T20:01:52.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: mon.smithi188@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-05-08T20:01:52.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: mon.smithi188@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-05-08T20:01:52.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: mon.smithi188@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-05-08T20:01:52.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: Deploying daemon mgr.smithi188.abfynx on smithi188 2024-05-08T20:01:52.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: from='client.? 172.21.15.188:0/841990849' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-08T20:01:52.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:52.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:52.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:52.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:52.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:52.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-08T20:01:52.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:01:52.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: Deploying daemon mon.smithi188 on smithi188 2024-05-08T20:01:52.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:52.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:52.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:52.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 ceph-mon[48761]: mon.smithi188@-1(synchronizing).paxosservice(auth 1..8) refresh upgraded, format 0 -> 3 2024-05-08T20:01:52.283 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 bash[48697]: 3d6556ae54b1cbf0166fce854d49491313bab45604077b364bdea46a314b2260 2024-05-08T20:01:52.283 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:52 smithi188 systemd[1]: Started Ceph mon.smithi188 for 526038a6-0d75-11ef-bc97-c7b262605968. 2024-05-08T20:01:52.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:51 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/45228432' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-08T20:01:53.701 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-08T20:01:53.702 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-08T20:01:53.927 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi188/config 2024-05-08T20:01:57.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: from='client.? 172.21.15.188:0/45228432' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-08T20:01:57.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:57.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi083"}]: dispatch 2024-05-08T20:01:57.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: mon.smithi083 calling monitor election 2024-05-08T20:01:57.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-05-08T20:01:57.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-05-08T20:01:57.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:57.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:01:57.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: from='mgr.? 172.21.15.188:0/2597019253' entity='mgr.smithi188.abfynx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi188.abfynx/crt"}]: dispatch 2024-05-08T20:01:57.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-05-08T20:01:57.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: mon.smithi188 calling monitor election 2024-05-08T20:01:57.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-05-08T20:01:57.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:57.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-05-08T20:01:57.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-05-08T20:01:57.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: mon.smithi083 is new leader, mons smithi083,smithi188 in quorum (ranks 0,1) 2024-05-08T20:01:57.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: monmap e2: 2 mons at {smithi083=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0],smithi188=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-08T20:01:57.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: fsmap 2024-05-08T20:01:57.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: osdmap e5: 0 total, 0 up, 0 in 2024-05-08T20:01:57.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: mgrmap e17: smithi083.fgcgxm(active, since 33s) 2024-05-08T20:01:57.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: overall HEALTH_OK 2024-05-08T20:01:57.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: Standby manager daemon smithi188.abfynx started 2024-05-08T20:01:57.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: from='mgr.? 172.21.15.188:0/2597019253' entity='mgr.smithi188.abfynx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-08T20:01:57.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:57.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: from='mgr.? 172.21.15.188:0/2597019253' entity='mgr.smithi188.abfynx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi188.abfynx/key"}]: dispatch 2024-05-08T20:01:57.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: from='mgr.? 172.21.15.188:0/2597019253' entity='mgr.smithi188.abfynx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-08T20:01:57.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:57.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:57.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:57.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:57.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:57.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi083"}]: dispatch 2024-05-08T20:01:57.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: mon.smithi083 calling monitor election 2024-05-08T20:01:57.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-05-08T20:01:57.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-05-08T20:01:57.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:57.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:01:57.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: from='mgr.? 172.21.15.188:0/2597019253' entity='mgr.smithi188.abfynx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi188.abfynx/crt"}]: dispatch 2024-05-08T20:01:57.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-05-08T20:01:57.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: mon.smithi188 calling monitor election 2024-05-08T20:01:57.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-05-08T20:01:57.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:57.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-05-08T20:01:57.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-05-08T20:01:57.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: mon.smithi083 is new leader, mons smithi083,smithi188 in quorum (ranks 0,1) 2024-05-08T20:01:57.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: monmap e2: 2 mons at {smithi083=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0],smithi188=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-08T20:01:57.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: fsmap 2024-05-08T20:01:57.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: osdmap e5: 0 total, 0 up, 0 in 2024-05-08T20:01:57.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: mgrmap e17: smithi083.fgcgxm(active, since 33s) 2024-05-08T20:01:57.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: overall HEALTH_OK 2024-05-08T20:01:57.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: Standby manager daemon smithi188.abfynx started 2024-05-08T20:01:57.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: from='mgr.? 172.21.15.188:0/2597019253' entity='mgr.smithi188.abfynx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-08T20:01:57.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:57.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: from='mgr.? 172.21.15.188:0/2597019253' entity='mgr.smithi188.abfynx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi188.abfynx/key"}]: dispatch 2024-05-08T20:01:57.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: from='mgr.? 172.21.15.188:0/2597019253' entity='mgr.smithi188.abfynx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-08T20:01:57.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:57.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:57.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:57.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:58.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:58 smithi188 ceph-mon[48761]: mgrmap e18: smithi083.fgcgxm(active, since 33s), standbys: smithi188.abfynx 2024-05-08T20:01:58.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:58 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mgr metadata", "who": "smithi188.abfynx", "id": "smithi188.abfynx"}]: dispatch 2024-05-08T20:01:58.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:58 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:01:58.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:58 smithi188 ceph-mon[48761]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:58.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:58 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-05-08T20:01:58.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:58 smithi083 ceph-mon[38695]: mgrmap e18: smithi083.fgcgxm(active, since 33s), standbys: smithi188.abfynx 2024-05-08T20:01:58.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:58 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mgr metadata", "who": "smithi188.abfynx", "id": "smithi188.abfynx"}]: dispatch 2024-05-08T20:01:58.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:58 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:01:58.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:58 smithi083 ceph-mon[38695]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:01:58.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:58 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-05-08T20:01:58.834 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T20:01:58.835 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":2,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","modified":"2024-05-08T20:01:52.008065Z","created":"2024-05-08T19:59:04.238534Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi188","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:3300","nonce":0},{"type":"v1","addr":"172.21.15.188:6789","nonce":0}]},"addr":"172.21.15.188:6789/0","public_addr":"172.21.15.188:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-08T20:01:58.837 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 2 2024-05-08T20:01:59.578 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-08T20:01:59.578 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph config generate-minimal-conf 2024-05-08T20:01:59.824 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:01:59.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:59.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:59 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/1212527873' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-08T20:01:59.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:59.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:01:59.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:01:59.934 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:01:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:02:00.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:00.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:59 smithi188 ceph-mon[48761]: from='client.? 172.21.15.188:0/1212527873' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-08T20:02:00.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:00.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:00.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:00.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:01:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:02:01.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:00 smithi188 ceph-mon[48761]: Updating smithi083:/etc/ceph/ceph.conf 2024-05-08T20:02:01.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:00 smithi188 ceph-mon[48761]: Updating smithi188:/etc/ceph/ceph.conf 2024-05-08T20:02:01.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:00 smithi188 ceph-mon[48761]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:01.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:00 smithi188 ceph-mon[48761]: Updating smithi083:/var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-08T20:02:01.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:00 smithi188 ceph-mon[48761]: Updating smithi188:/var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-08T20:02:01.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:00 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:01.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:00 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:01.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:00 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:01.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:00 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:01.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:00 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:01.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:00 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-08T20:02:01.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:00 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-08T20:02:01.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:00 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:01.123 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:00 smithi083 ceph-mon[38695]: Updating smithi083:/etc/ceph/ceph.conf 2024-05-08T20:02:01.123 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:00 smithi083 ceph-mon[38695]: Updating smithi188:/etc/ceph/ceph.conf 2024-05-08T20:02:01.123 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:00 smithi083 ceph-mon[38695]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:01.123 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:00 smithi083 ceph-mon[38695]: Updating smithi083:/var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-08T20:02:01.123 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:00 smithi083 ceph-mon[38695]: Updating smithi188:/var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-08T20:02:01.123 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:00 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:01.124 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:00 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:01.124 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:00 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:01.124 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:00 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:01.124 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:00 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:01.124 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:00 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-08T20:02:01.124 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:00 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-08T20:02:01.124 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:00 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:02.081 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:01 smithi083 ceph-mon[38695]: Reconfiguring mon.smithi083 (unknown last config time)... 2024-05-08T20:02:02.082 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:01 smithi083 ceph-mon[38695]: Reconfiguring daemon mon.smithi083 on smithi083 2024-05-08T20:02:02.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:01 smithi188 ceph-mon[48761]: Reconfiguring mon.smithi083 (unknown last config time)... 2024-05-08T20:02:02.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:01 smithi188 ceph-mon[48761]: Reconfiguring daemon mon.smithi083 on smithi083 2024-05-08T20:02:02.846 INFO:teuthology.orchestra.run.smithi083.stdout:# minimal ceph.conf for 526038a6-0d75-11ef-bc97-c7b262605968 2024-05-08T20:02:02.846 INFO:teuthology.orchestra.run.smithi083.stdout:[global] 2024-05-08T20:02:02.847 INFO:teuthology.orchestra.run.smithi083.stdout: fsid = 526038a6-0d75-11ef-bc97-c7b262605968 2024-05-08T20:02:02.847 INFO:teuthology.orchestra.run.smithi083.stdout: mon_host = [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] 2024-05-08T20:02:03.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:02 smithi083 ceph-mon[38695]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:03.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:03.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:03.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi083.fgcgxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-08T20:02:03.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-08T20:02:03.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:03.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:02 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2859863690' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:03.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:02 smithi188 ceph-mon[48761]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:03.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:03.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:03.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi083.fgcgxm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-08T20:02:03.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-08T20:02:03.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:03.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:02 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/2859863690' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:03.710 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-08T20:02:03.711 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-05-08T20:02:03.711 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.conf 2024-05-08T20:02:03.740 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-05-08T20:02:03.740 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-08T20:02:03.808 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-05-08T20:02:03.808 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.conf 2024-05-08T20:02:03.845 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-05-08T20:02:03.845 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-08T20:02:03.916 INFO:tasks.cephadm:Deploying OSDs... 2024-05-08T20:02:03.916 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-05-08T20:02:03.916 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/scratch_devs of=/dev/stdout 2024-05-08T20:02:03.934 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-08T20:02:03.934 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_1 2024-05-08T20:02:03.992 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-08T20:02:03.992 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T20:02:03.992 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 6h/6d Inode: 162685 Links: 1 2024-05-08T20:02:03.992 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T20:02:03.992 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T20:02:03.992 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-05-08 20:01:15.196654786 +0000 2024-05-08T20:02:03.992 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-05-08 19:54:06.162995577 +0000 2024-05-08T20:02:03.993 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-05-08 19:54:06.162995577 +0000 2024-05-08T20:02:03.993 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: - 2024-05-08T20:02:03.993 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-08T20:02:04.057 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-05-08T20:02:04.058 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-05-08T20:02:04.058 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000260756 s, 2.0 MB/s 2024-05-08T20:02:04.059 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-08T20:02:04.118 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_2 2024-05-08T20:02:04.176 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-08T20:02:04.176 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T20:02:04.176 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 6h/6d Inode: 169399 Links: 1 2024-05-08T20:02:04.176 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T20:02:04.176 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T20:02:04.176 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-05-08 20:01:15.196654786 +0000 2024-05-08T20:02:04.176 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-05-08 19:54:06.499987678 +0000 2024-05-08T20:02:04.176 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-05-08 19:54:06.499987678 +0000 2024-05-08T20:02:04.176 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: - 2024-05-08T20:02:04.176 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-08T20:02:04.241 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:03 smithi083 ceph-mon[38695]: Reconfiguring mgr.smithi083.fgcgxm (unknown last config time)... 2024-05-08T20:02:04.241 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:03 smithi083 ceph-mon[38695]: Reconfiguring daemon mgr.smithi083.fgcgxm on smithi083 2024-05-08T20:02:04.243 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-05-08T20:02:04.243 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-05-08T20:02:04.243 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000235529 s, 2.2 MB/s 2024-05-08T20:02:04.244 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-08T20:02:04.304 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_3 2024-05-08T20:02:04.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:03 smithi188 ceph-mon[48761]: Reconfiguring mgr.smithi083.fgcgxm (unknown last config time)... 2024-05-08T20:02:04.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:03 smithi188 ceph-mon[48761]: Reconfiguring daemon mgr.smithi083.fgcgxm on smithi083 2024-05-08T20:02:04.362 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-08T20:02:04.362 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T20:02:04.363 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 6h/6d Inode: 171053 Links: 1 2024-05-08T20:02:04.363 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T20:02:04.363 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T20:02:04.363 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-05-08 20:01:15.196654786 +0000 2024-05-08T20:02:04.363 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-05-08 19:54:06.853979381 +0000 2024-05-08T20:02:04.363 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-05-08 19:54:06.853979381 +0000 2024-05-08T20:02:04.363 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: - 2024-05-08T20:02:04.363 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-08T20:02:04.428 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-05-08T20:02:04.428 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-05-08T20:02:04.428 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000187972 s, 2.7 MB/s 2024-05-08T20:02:04.429 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-08T20:02:04.488 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_4 2024-05-08T20:02:04.546 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-08T20:02:04.546 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T20:02:04.547 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 6h/6d Inode: 173087 Links: 1 2024-05-08T20:02:04.547 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T20:02:04.547 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T20:02:04.547 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-05-08 20:01:15.196654786 +0000 2024-05-08T20:02:04.547 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-05-08 19:54:07.203971178 +0000 2024-05-08T20:02:04.547 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-05-08 19:54:07.203971178 +0000 2024-05-08T20:02:04.547 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: - 2024-05-08T20:02:04.547 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-08T20:02:04.612 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-05-08T20:02:04.613 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-05-08T20:02:04.613 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000212098 s, 2.4 MB/s 2024-05-08T20:02:04.614 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-08T20:02:04.673 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-05-08T20:02:04.673 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/scratch_devs of=/dev/stdout 2024-05-08T20:02:04.695 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-08T20:02:04.695 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_1 2024-05-08T20:02:04.756 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-08T20:02:04.756 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T20:02:04.757 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 167302 Links: 1 2024-05-08T20:02:04.757 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T20:02:04.757 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T20:02:04.757 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-05-08 20:01:51.278503060 +0000 2024-05-08T20:02:04.757 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-05-08 19:54:06.285103514 +0000 2024-05-08T20:02:04.757 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-05-08 19:54:06.285103514 +0000 2024-05-08T20:02:04.757 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2024-05-08T20:02:04.757 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-08T20:02:04.827 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-05-08T20:02:04.827 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-05-08T20:02:04.827 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000239831 s, 2.1 MB/s 2024-05-08T20:02:04.828 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-08T20:02:04.890 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_2 2024-05-08T20:02:04.950 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-08T20:02:04.950 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T20:02:04.950 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 165706 Links: 1 2024-05-08T20:02:04.950 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T20:02:04.950 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T20:02:04.950 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-05-08 20:01:51.278503060 +0000 2024-05-08T20:02:04.950 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-05-08 19:54:06.616097436 +0000 2024-05-08T20:02:04.951 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-05-08 19:54:06.616097436 +0000 2024-05-08T20:02:04.951 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2024-05-08T20:02:04.951 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-08T20:02:05.018 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-05-08T20:02:05.019 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-05-08T20:02:05.019 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000235456 s, 2.2 MB/s 2024-05-08T20:02:05.020 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-08T20:02:05.080 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_3 2024-05-08T20:02:05.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:04 smithi188 ceph-mon[48761]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:05.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:04 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:05.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:04 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:05.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:04 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi083", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-05-08T20:02:05.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:04 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:05.100 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-08T20:02:05.100 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T20:02:05.100 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 171332 Links: 1 2024-05-08T20:02:05.100 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T20:02:05.101 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T20:02:05.101 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-05-08 20:01:51.278503060 +0000 2024-05-08T20:02:05.101 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-05-08 19:54:06.957091173 +0000 2024-05-08T20:02:05.101 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-05-08 19:54:06.957091173 +0000 2024-05-08T20:02:05.101 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2024-05-08T20:02:05.101 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-08T20:02:05.167 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-05-08T20:02:05.167 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-05-08T20:02:05.167 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000188129 s, 2.7 MB/s 2024-05-08T20:02:05.170 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-08T20:02:05.229 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_4 2024-05-08T20:02:05.251 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:04 smithi083 ceph-mon[38695]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:05.251 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:04 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:05.251 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:04 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:05.251 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:04 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi083", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-05-08T20:02:05.251 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:04 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:05.287 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-08T20:02:05.287 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T20:02:05.287 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 163783 Links: 1 2024-05-08T20:02:05.287 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T20:02:05.287 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T20:02:05.287 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-05-08 20:01:51.278503060 +0000 2024-05-08T20:02:05.287 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-05-08 19:54:07.335084230 +0000 2024-05-08T20:02:05.287 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-05-08 19:54:07.335084230 +0000 2024-05-08T20:02:05.287 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2024-05-08T20:02:05.288 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-08T20:02:05.353 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-05-08T20:02:05.353 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-05-08T20:02:05.353 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000230808 s, 2.2 MB/s 2024-05-08T20:02:05.354 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-08T20:02:05.412 INFO:tasks.cephadm:Deploying osd.0 on smithi083 with /dev/vg_nvme/lv_4... 2024-05-08T20:02:05.413 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-08T20:02:06.069 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:02:06.294 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:05 smithi083 ceph-mon[38695]: Reconfiguring ceph-exporter.smithi083 (monmap changed)... 2024-05-08T20:02:06.294 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:05 smithi083 ceph-mon[38695]: Reconfiguring daemon ceph-exporter.smithi083 on smithi083 2024-05-08T20:02:06.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:05 smithi188 ceph-mon[48761]: Reconfiguring ceph-exporter.smithi083 (monmap changed)... 2024-05-08T20:02:06.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:05 smithi188 ceph-mon[48761]: Reconfiguring daemon ceph-exporter.smithi083 on smithi083 2024-05-08T20:02:07.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:07 smithi083 ceph-mon[38695]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:07.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:07 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:07.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:07 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:07.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:07 smithi083 ceph-mon[38695]: Reconfiguring crash.smithi083 (monmap changed)... 2024-05-08T20:02:07.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:07 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi083", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-05-08T20:02:07.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:07 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:07.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:07 smithi083 ceph-mon[38695]: Reconfiguring daemon crash.smithi083 on smithi083 2024-05-08T20:02:07.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:07 smithi188 ceph-mon[48761]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:07.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:07 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:07.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:07 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:07.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:07 smithi188 ceph-mon[48761]: Reconfiguring crash.smithi083 (monmap changed)... 2024-05-08T20:02:07.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:07 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi083", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-05-08T20:02:07.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:07 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:07.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:07 smithi188 ceph-mon[48761]: Reconfiguring daemon crash.smithi083 on smithi083 2024-05-08T20:02:09.237 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-08T20:02:09.237 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-08T20:02:09.237 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10+0 records in 2024-05-08T20:02:09.237 INFO:teuthology.orchestra.run.smithi083.stderr:10+0 records out 2024-05-08T20:02:09.237 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109841 s, 955 MB/s 2024-05-08T20:02:09.237 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping successful for: 2024-05-08T20:02:09.258 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:09 smithi083 ceph-mon[38695]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:09.259 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:09.259 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:09.259 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:02:09.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:09 smithi188 ceph-mon[48761]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:09.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:09.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:09.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:02:10.283 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:10 smithi083 ceph-mon[38695]: Reconfiguring alertmanager.smithi083 (dependencies changed)... 2024-05-08T20:02:10.283 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:10 smithi083 ceph-mon[38695]: Reconfiguring daemon alertmanager.smithi083 on smithi083 2024-05-08T20:02:10.303 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi083:vg_nvme/lv_4 2024-05-08T20:02:10.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:10 smithi188 ceph-mon[48761]: Reconfiguring alertmanager.smithi083 (dependencies changed)... 2024-05-08T20:02:10.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:10 smithi188 ceph-mon[48761]: Reconfiguring daemon alertmanager.smithi083 on smithi083 2024-05-08T20:02:11.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:11 smithi083 ceph-mon[38695]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:11.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:11 smithi188 ceph-mon[48761]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:11.833 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:02:13.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:13 smithi188 ceph-mon[48761]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:13.406 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:13 smithi083 ceph-mon[38695]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:15.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:15 smithi083 ceph-mon[38695]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:15.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:15 smithi188 ceph-mon[48761]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:16.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:16 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-08T20:02:16.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:16 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-08T20:02:16.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:16 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:16.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:16 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:16.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:16 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:16.411 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:16 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-08T20:02:16.411 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:16 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-08T20:02:16.412 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:16 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:16.412 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:16 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:16.412 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:16 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:17.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:17 smithi188 ceph-mon[48761]: from='client.14248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:02:17.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:17 smithi188 ceph-mon[48761]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:17.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:17 smithi188 ceph-mon[48761]: Reconfiguring grafana.smithi083 (dependencies changed)... 2024-05-08T20:02:17.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:17 smithi188 ceph-mon[48761]: Reconfiguring daemon grafana.smithi083 on smithi083 2024-05-08T20:02:17.344 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:17 smithi083 ceph-mon[38695]: from='client.14248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:02:17.344 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:17 smithi083 ceph-mon[38695]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:17.345 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:17 smithi083 ceph-mon[38695]: Reconfiguring grafana.smithi083 (dependencies changed)... 2024-05-08T20:02:17.345 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:17 smithi083 ceph-mon[38695]: Reconfiguring daemon grafana.smithi083 on smithi083 2024-05-08T20:02:19.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:19 smithi083 ceph-mon[38695]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:19.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:19 smithi188 ceph-mon[48761]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:21.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:21 smithi083 ceph-mon[38695]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:21.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:21 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:21.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:21 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:21.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:21 smithi188 ceph-mon[48761]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:21.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:21 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:21.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:21 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:22.186 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:22 smithi083 ceph-mon[38695]: Reconfiguring prometheus.smithi083 (dependencies changed)... 2024-05-08T20:02:22.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:22 smithi188 ceph-mon[48761]: Reconfiguring prometheus.smithi083 (dependencies changed)... 2024-05-08T20:02:23.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:23 smithi188 ceph-mon[48761]: Reconfiguring daemon prometheus.smithi083 on smithi083 2024-05-08T20:02:23.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:23 smithi188 ceph-mon[48761]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:23.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:23 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/2338231990' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78051d3d-3644-4876-af5f-8c956a8491bc"}]: dispatch 2024-05-08T20:02:23.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:23 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/2338231990' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "78051d3d-3644-4876-af5f-8c956a8491bc"}]': finished 2024-05-08T20:02:23.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:23 smithi188 ceph-mon[48761]: osdmap e6: 1 total, 0 up, 1 in 2024-05-08T20:02:23.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-08T20:02:23.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:23 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/1112905020' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-08T20:02:23.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:23 smithi083 ceph-mon[38695]: Reconfiguring daemon prometheus.smithi083 on smithi083 2024-05-08T20:02:23.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:23 smithi083 ceph-mon[38695]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:23.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:23 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2338231990' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78051d3d-3644-4876-af5f-8c956a8491bc"}]: dispatch 2024-05-08T20:02:23.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:23 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2338231990' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "78051d3d-3644-4876-af5f-8c956a8491bc"}]': finished 2024-05-08T20:02:23.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:23 smithi083 ceph-mon[38695]: osdmap e6: 1 total, 0 up, 1 in 2024-05-08T20:02:23.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-08T20:02:23.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:23 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/1112905020' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-08T20:02:24.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:02:24.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:24 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:02:25.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:25 smithi188 ceph-mon[48761]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:25.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:25 smithi083 ceph-mon[38695]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:27.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:27 smithi188 ceph-mon[48761]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:27.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:27 smithi083 ceph-mon[38695]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:29.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:29 smithi083 ceph-mon[38695]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:29.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:29 smithi188 ceph-mon[48761]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:31.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:31 smithi083 ceph-mon[38695]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:31.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:31 smithi188 ceph-mon[48761]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:32.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:32 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:32.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:32 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:32.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:32 smithi083 ceph-mon[38695]: Reconfiguring ceph-exporter.smithi188 (monmap changed)... 2024-05-08T20:02:32.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:32 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi188", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-05-08T20:02:32.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:32 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:32.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:32 smithi083 ceph-mon[38695]: Reconfiguring daemon ceph-exporter.smithi188 on smithi188 2024-05-08T20:02:32.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:32 smithi083 ceph-mon[38695]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:32.832 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:32 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:32.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:32 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:32.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:32 smithi188 ceph-mon[48761]: Reconfiguring ceph-exporter.smithi188 (monmap changed)... 2024-05-08T20:02:32.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:32 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi188", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-05-08T20:02:32.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:32 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:32.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:32 smithi188 ceph-mon[48761]: Reconfiguring daemon ceph-exporter.smithi188 on smithi188 2024-05-08T20:02:32.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:32 smithi188 ceph-mon[48761]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:34.264 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:34 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:34.264 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:34 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:34.264 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:34 smithi083 ceph-mon[38695]: Reconfiguring crash.smithi188 (monmap changed)... 2024-05-08T20:02:34.264 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:34 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi188", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-05-08T20:02:34.264 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:34 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:34.265 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:34 smithi083 ceph-mon[38695]: Reconfiguring daemon crash.smithi188 on smithi188 2024-05-08T20:02:34.265 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:34 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-08T20:02:34.265 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:34 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:34.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:34 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:34.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:34 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:34.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:34 smithi188 ceph-mon[48761]: Reconfiguring crash.smithi188 (monmap changed)... 2024-05-08T20:02:34.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:34 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi188", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-05-08T20:02:34.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:34 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:34.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:34 smithi188 ceph-mon[48761]: Reconfiguring daemon crash.smithi188 on smithi188 2024-05-08T20:02:34.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:34 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-08T20:02:34.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:34 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:35.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:35 smithi083 ceph-mon[38695]: Deploying daemon osd.0 on smithi083 2024-05-08T20:02:35.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:35 smithi083 ceph-mon[38695]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:35.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:35.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:35.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi188.abfynx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-08T20:02:35.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-08T20:02:35.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:35.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:35 smithi188 ceph-mon[48761]: Deploying daemon osd.0 on smithi083 2024-05-08T20:02:35.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:35 smithi188 ceph-mon[48761]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:35.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:35.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:35.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi188.abfynx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-08T20:02:35.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-08T20:02:35.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:36.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:36 smithi083 ceph-mon[38695]: Reconfiguring mgr.smithi188.abfynx (monmap changed)... 2024-05-08T20:02:36.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:36 smithi083 ceph-mon[38695]: Reconfiguring daemon mgr.smithi188.abfynx on smithi188 2024-05-08T20:02:36.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:36 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:36.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:36 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:36.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:36 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-08T20:02:36.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:36 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-08T20:02:36.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:36 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:36.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:36 smithi188 ceph-mon[48761]: Reconfiguring mgr.smithi188.abfynx (monmap changed)... 2024-05-08T20:02:36.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:36 smithi188 ceph-mon[48761]: Reconfiguring daemon mgr.smithi188.abfynx on smithi188 2024-05-08T20:02:36.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:36 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:36.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:36 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:36.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:36 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-08T20:02:36.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:36 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-08T20:02:36.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:36 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:37.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:37 smithi188 ceph-mon[48761]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:37.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:37 smithi188 ceph-mon[48761]: Reconfiguring mon.smithi188 (monmap changed)... 2024-05-08T20:02:37.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:37 smithi188 ceph-mon[48761]: Reconfiguring daemon mon.smithi188 on smithi188 2024-05-08T20:02:37.345 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:37 smithi083 ceph-mon[38695]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:37.345 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:37 smithi083 ceph-mon[38695]: Reconfiguring mon.smithi188 (monmap changed)... 2024-05-08T20:02:37.345 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:37 smithi083 ceph-mon[38695]: Reconfiguring daemon mon.smithi188 on smithi188 2024-05-08T20:02:38.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:38.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:38.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-08T20:02:38.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:38 smithi188 ceph-mon[48761]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-08T20:02:38.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi083.front.sepia.ceph.com:9093"}]: dispatch 2024-05-08T20:02:38.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:38 smithi188 ceph-mon[48761]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi083.front.sepia.ceph.com:9093"}]: dispatch 2024-05-08T20:02:38.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:38.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-08T20:02:38.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:38 smithi188 ceph-mon[48761]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-08T20:02:38.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi083.front.sepia.ceph.com:3000"}]: dispatch 2024-05-08T20:02:38.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:38 smithi188 ceph-mon[48761]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi083.front.sepia.ceph.com:3000"}]: dispatch 2024-05-08T20:02:38.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:38.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-08T20:02:38.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:38 smithi188 ceph-mon[48761]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-08T20:02:38.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi083.front.sepia.ceph.com:9095"}]: dispatch 2024-05-08T20:02:38.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:38 smithi188 ceph-mon[48761]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi083.front.sepia.ceph.com:9095"}]: dispatch 2024-05-08T20:02:38.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:38.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:02:38.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:38 smithi188 ceph-mon[48761]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:38.414 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:38.414 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:38.414 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-08T20:02:38.414 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:38 smithi083 ceph-mon[38695]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-08T20:02:38.414 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi083.front.sepia.ceph.com:9093"}]: dispatch 2024-05-08T20:02:38.414 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:38 smithi083 ceph-mon[38695]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi083.front.sepia.ceph.com:9093"}]: dispatch 2024-05-08T20:02:38.414 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:38.414 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-08T20:02:38.414 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:38 smithi083 ceph-mon[38695]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-08T20:02:38.414 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi083.front.sepia.ceph.com:3000"}]: dispatch 2024-05-08T20:02:38.415 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:38 smithi083 ceph-mon[38695]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi083.front.sepia.ceph.com:3000"}]: dispatch 2024-05-08T20:02:38.415 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:38.415 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-08T20:02:38.415 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:38 smithi083 ceph-mon[38695]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-08T20:02:38.415 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi083.front.sepia.ceph.com:9095"}]: dispatch 2024-05-08T20:02:38.415 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:38 smithi083 ceph-mon[38695]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi083.front.sepia.ceph.com:9095"}]: dispatch 2024-05-08T20:02:38.415 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:38.415 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:02:38.415 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:38 smithi083 ceph-mon[38695]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:39.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:39.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:39.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:02:39.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:39.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:40.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:40.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:40.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:02:40.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:40.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:40.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:40 smithi083 ceph-mon[38695]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:40.976 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:40 smithi188 ceph-mon[48761]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:42.172 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:41 smithi083 ceph-mon[38695]: from='osd.0 [v2:172.21.15.83:6802/345419579,v1:172.21.15.83:6803/345419579]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-08T20:02:42.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:41 smithi188 ceph-mon[48761]: from='osd.0 [v2:172.21.15.83:6802/345419579,v1:172.21.15.83:6803/345419579]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-08T20:02:42.527 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 0 on host 'smithi083' 2024-05-08T20:02:43.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:42 smithi083 ceph-mon[38695]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:43.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:42 smithi083 ceph-mon[38695]: from='osd.0 [v2:172.21.15.83:6802/345419579,v1:172.21.15.83:6803/345419579]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-08T20:02:43.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:42 smithi083 ceph-mon[38695]: osdmap e7: 1 total, 0 up, 1 in 2024-05-08T20:02:43.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:42 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-08T20:02:43.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:42 smithi083 ceph-mon[38695]: from='osd.0 [v2:172.21.15.83:6802/345419579,v1:172.21.15.83:6803/345419579]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-05-08T20:02:43.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:42 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:43.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:42 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:43.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:42 smithi083 ceph-mon[38695]: from='osd.0 [v2:172.21.15.83:6802/345419579,v1:172.21.15.83:6803/345419579]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-05-08T20:02:43.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:42 smithi083 ceph-mon[38695]: osdmap e8: 1 total, 0 up, 1 in 2024-05-08T20:02:43.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:42 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-08T20:02:43.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:42 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-08T20:02:43.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:42 smithi188 ceph-mon[48761]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:43.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:42 smithi188 ceph-mon[48761]: from='osd.0 [v2:172.21.15.83:6802/345419579,v1:172.21.15.83:6803/345419579]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-08T20:02:43.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:42 smithi188 ceph-mon[48761]: osdmap e7: 1 total, 0 up, 1 in 2024-05-08T20:02:43.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:42 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-08T20:02:43.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:42 smithi188 ceph-mon[48761]: from='osd.0 [v2:172.21.15.83:6802/345419579,v1:172.21.15.83:6803/345419579]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-05-08T20:02:43.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:42 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:43.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:42 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:43.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:42 smithi188 ceph-mon[48761]: from='osd.0 [v2:172.21.15.83:6802/345419579,v1:172.21.15.83:6803/345419579]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-05-08T20:02:43.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:42 smithi188 ceph-mon[48761]: osdmap e8: 1 total, 0 up, 1 in 2024-05-08T20:02:43.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:42 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-08T20:02:43.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:42 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-08T20:02:44.039 DEBUG:teuthology.orchestra.run.smithi083:osd.0> sudo journalctl -f -n 0 -u ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.0.service 2024-05-08T20:02:44.041 INFO:tasks.cephadm:Deploying osd.1 on smithi083 with /dev/vg_nvme/lv_3... 2024-05-08T20:02:44.041 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-08T20:02:44.075 INFO:journalctl@ceph.osd.0.smithi083.stdout:-- Logs begin at Wed 2024-05-08 19:47:49 UTC. -- 2024-05-08T20:02:44.582 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:44 smithi188 ceph-mon[48761]: from='osd.0 [v2:172.21.15.83:6802/345419579,v1:172.21.15.83:6803/345419579]' entity='osd.0' 2024-05-08T20:02:44.583 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:44 smithi188 ceph-mon[48761]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:44.583 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:44 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-08T20:02:44.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:44 smithi083 ceph-mon[38695]: from='osd.0 [v2:172.21.15.83:6802/345419579,v1:172.21.15.83:6803/345419579]' entity='osd.0' 2024-05-08T20:02:44.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:44 smithi083 ceph-mon[38695]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-08T20:02:44.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:44 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-08T20:02:44.747 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:02:45.583 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:45 smithi188 ceph-mon[48761]: purged_snaps scrub starts 2024-05-08T20:02:45.583 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:45 smithi188 ceph-mon[48761]: purged_snaps scrub ok 2024-05-08T20:02:45.584 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:45 smithi188 ceph-mon[48761]: osd.0 [v2:172.21.15.83:6802/345419579,v1:172.21.15.83:6803/345419579] boot 2024-05-08T20:02:45.584 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:45 smithi188 ceph-mon[48761]: osdmap e9: 1 total, 1 up, 1 in 2024-05-08T20:02:45.584 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:45 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-08T20:02:45.680 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:45 smithi083 ceph-mon[38695]: purged_snaps scrub starts 2024-05-08T20:02:45.680 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:45 smithi083 ceph-mon[38695]: purged_snaps scrub ok 2024-05-08T20:02:45.680 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:45 smithi083 ceph-mon[38695]: osd.0 [v2:172.21.15.83:6802/345419579,v1:172.21.15.83:6803/345419579] boot 2024-05-08T20:02:45.680 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:45 smithi083 ceph-mon[38695]: osdmap e9: 1 total, 1 up, 1 in 2024-05-08T20:02:45.680 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:45 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-08T20:02:46.582 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:46 smithi188 ceph-mon[48761]: osdmap e10: 1 total, 1 up, 1 in 2024-05-08T20:02:46.583 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:46 smithi188 ceph-mon[48761]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:02:46.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:46 smithi083 ceph-mon[38695]: osdmap e10: 1 total, 1 up, 1 in 2024-05-08T20:02:46.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:46 smithi083 ceph-mon[38695]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:02:47.503 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-08T20:02:47.503 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-08T20:02:47.503 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10+0 records in 2024-05-08T20:02:47.503 INFO:teuthology.orchestra.run.smithi083.stderr:10+0 records out 2024-05-08T20:02:47.503 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012958 s, 809 MB/s 2024-05-08T20:02:47.504 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping successful for: 2024-05-08T20:02:47.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:47 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:47.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:47 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:48.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:47 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:48.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:47 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:48.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:48 smithi083 ceph-mon[38695]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:02:49.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:48 smithi188 ceph-mon[48761]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:02:49.209 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi083:vg_nvme/lv_3 2024-05-08T20:02:49.742 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:02:51.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:50 smithi188 ceph-mon[48761]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:02:51.143 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:50 smithi083 ceph-mon[38695]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:02:52.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:52 smithi188 ceph-mon[48761]: Detected new or changed devices on smithi083 2024-05-08T20:02:52.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:52 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:52.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:52 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:52.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:52 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:02:52.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:52 smithi188 ceph-mon[48761]: Adjusting osd_memory_target on smithi083 to 11874M 2024-05-08T20:02:52.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:52 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:52.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:52 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:52.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:52 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:02:52.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:52 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:52.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:52 smithi188 ceph-mon[48761]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:02:52.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:52 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:02:52.834 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:52 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-08T20:02:52.834 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:52 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-08T20:02:52.834 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:52 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:52.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:52 smithi083 ceph-mon[38695]: Detected new or changed devices on smithi083 2024-05-08T20:02:52.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:52 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:52.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:52 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:52.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:52 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:02:52.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:52 smithi083 ceph-mon[38695]: Adjusting osd_memory_target on smithi083 to 11874M 2024-05-08T20:02:52.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:52 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:52.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:52 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:52.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:52 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:02:52.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:52 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:52.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:52 smithi083 ceph-mon[38695]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:02:52.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:52 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:02:52.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:52 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-08T20:02:52.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:52 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-08T20:02:52.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:52 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:53.886 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:53 smithi083 ceph-mon[38695]: from='client.14266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:02:54.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:53 smithi188 ceph-mon[48761]: from='client.14266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:02:55.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:54 smithi188 ceph-mon[48761]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:02:55.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:02:55.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:55.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:55.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:54 smithi083 ceph-mon[38695]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:02:55.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:02:55.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:55.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:55.996 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:55 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/570104226' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc62b8e9-e30e-41f1-aaee-4c4e356be487"}]: dispatch 2024-05-08T20:02:55.996 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:55 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/570104226' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc62b8e9-e30e-41f1-aaee-4c4e356be487"}]': finished 2024-05-08T20:02:55.996 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:55 smithi188 ceph-mon[48761]: osdmap e11: 2 total, 1 up, 2 in 2024-05-08T20:02:55.996 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:55 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-08T20:02:56.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:55 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/570104226' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc62b8e9-e30e-41f1-aaee-4c4e356be487"}]: dispatch 2024-05-08T20:02:56.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:55 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/570104226' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc62b8e9-e30e-41f1-aaee-4c4e356be487"}]': finished 2024-05-08T20:02:56.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:55 smithi083 ceph-mon[38695]: osdmap e11: 2 total, 1 up, 2 in 2024-05-08T20:02:56.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:55 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-08T20:02:56.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:56 smithi083 ceph-mon[38695]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:02:56.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:56 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/1757079349' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-08T20:02:57.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:56 smithi188 ceph-mon[48761]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:02:57.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:56 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/1757079349' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-08T20:02:59.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:58 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:59.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:58 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:59.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:58 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:59.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:58 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:02:59.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:58 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:59.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:02:58 smithi188 ceph-mon[48761]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:02:59.084 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:58 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:59.085 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:58 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:59.085 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:58 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:02:59.085 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:58 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:02:59.085 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:58 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:02:59.085 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:02:58 smithi083 ceph-mon[38695]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:03:01.020 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:00 smithi083 ceph-mon[38695]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:03:01.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:00 smithi188 ceph-mon[48761]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:03:02.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:02 smithi083 ceph-mon[38695]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:03:02.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-08T20:03:02.902 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:03:03.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:02 smithi188 ceph-mon[48761]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:03:03.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-08T20:03:03.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:03:04.050 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:03 smithi083 ceph-mon[38695]: Deploying daemon osd.1 on smithi083 2024-05-08T20:03:04.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:03 smithi188 ceph-mon[48761]: Deploying daemon osd.1 on smithi083 2024-05-08T20:03:05.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:04 smithi188 ceph-mon[48761]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:03:05.093 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:04 smithi083 ceph-mon[38695]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:03:07.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:06 smithi083 ceph-mon[38695]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:03:07.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:06 smithi188 ceph-mon[48761]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:03:08.883 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:08.883 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:08.883 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:03:08.883 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:08 smithi083 ceph-mon[38695]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:03:09.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:09.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:09.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:03:09.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:08 smithi188 ceph-mon[48761]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:03:10.018 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:03:10.018 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:10.018 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:10.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:03:10.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:10.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:11.015 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:10 smithi188 ceph-mon[48761]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:03:11.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:10 smithi083 ceph-mon[38695]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:03:11.610 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 1 on host 'smithi083' 2024-05-08T20:03:11.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:11 smithi083 ceph-mon[38695]: from='osd.1 [v2:172.21.15.83:6810/1992633959,v1:172.21.15.83:6811/1992633959]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-08T20:03:11.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:11 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:11.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:11 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:12.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:11 smithi188 ceph-mon[48761]: from='osd.1 [v2:172.21.15.83:6810/1992633959,v1:172.21.15.83:6811/1992633959]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-08T20:03:12.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:11 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:12.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:11 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:13.032 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:12 smithi083 ceph-mon[38695]: from='osd.1 [v2:172.21.15.83:6810/1992633959,v1:172.21.15.83:6811/1992633959]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-08T20:03:13.032 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:12 smithi083 ceph-mon[38695]: osdmap e12: 2 total, 1 up, 2 in 2024-05-08T20:03:13.032 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:12 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-08T20:03:13.032 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:12 smithi083 ceph-mon[38695]: from='osd.1 [v2:172.21.15.83:6810/1992633959,v1:172.21.15.83:6811/1992633959]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-05-08T20:03:13.032 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:12 smithi083 ceph-mon[38695]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:03:13.032 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:12 smithi083 ceph-mon[38695]: from='osd.1 [v2:172.21.15.83:6810/1992633959,v1:172.21.15.83:6811/1992633959]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-05-08T20:03:13.032 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:12 smithi083 ceph-mon[38695]: osdmap e13: 2 total, 1 up, 2 in 2024-05-08T20:03:13.033 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:12 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-08T20:03:13.033 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:12 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-08T20:03:13.054 DEBUG:teuthology.orchestra.run.smithi083:osd.1> sudo journalctl -f -n 0 -u ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.1.service 2024-05-08T20:03:13.057 INFO:tasks.cephadm:Deploying osd.2 on smithi083 with /dev/vg_nvme/lv_2... 2024-05-08T20:03:13.057 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-08T20:03:13.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:12 smithi188 ceph-mon[48761]: from='osd.1 [v2:172.21.15.83:6810/1992633959,v1:172.21.15.83:6811/1992633959]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-08T20:03:13.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:12 smithi188 ceph-mon[48761]: osdmap e12: 2 total, 1 up, 2 in 2024-05-08T20:03:13.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:12 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-08T20:03:13.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:12 smithi188 ceph-mon[48761]: from='osd.1 [v2:172.21.15.83:6810/1992633959,v1:172.21.15.83:6811/1992633959]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-05-08T20:03:13.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:12 smithi188 ceph-mon[48761]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:03:13.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:12 smithi188 ceph-mon[48761]: from='osd.1 [v2:172.21.15.83:6810/1992633959,v1:172.21.15.83:6811/1992633959]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-05-08T20:03:13.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:12 smithi188 ceph-mon[48761]: osdmap e13: 2 total, 1 up, 2 in 2024-05-08T20:03:13.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:12 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-08T20:03:13.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:12 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-08T20:03:13.087 INFO:journalctl@ceph.osd.1.smithi083.stdout:-- Logs begin at Wed 2024-05-08 19:47:49 UTC. -- 2024-05-08T20:03:13.497 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:03:14.010 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:13 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-08T20:03:14.010 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:13 smithi083 ceph-mon[38695]: osd.1 [v2:172.21.15.83:6810/1992633959,v1:172.21.15.83:6811/1992633959] boot 2024-05-08T20:03:14.010 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:13 smithi083 ceph-mon[38695]: osdmap e14: 2 total, 2 up, 2 in 2024-05-08T20:03:14.010 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:13 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-08T20:03:14.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:13 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-08T20:03:14.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:13 smithi188 ceph-mon[48761]: osd.1 [v2:172.21.15.83:6810/1992633959,v1:172.21.15.83:6811/1992633959] boot 2024-05-08T20:03:14.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:13 smithi188 ceph-mon[48761]: osdmap e14: 2 total, 2 up, 2 in 2024-05-08T20:03:14.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:13 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-08T20:03:15.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:14 smithi188 ceph-mon[48761]: purged_snaps scrub starts 2024-05-08T20:03:15.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:14 smithi188 ceph-mon[48761]: purged_snaps scrub ok 2024-05-08T20:03:15.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:14 smithi188 ceph-mon[48761]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:03:15.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:14 smithi083 ceph-mon[38695]: purged_snaps scrub starts 2024-05-08T20:03:15.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:14 smithi083 ceph-mon[38695]: purged_snaps scrub ok 2024-05-08T20:03:15.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:14 smithi083 ceph-mon[38695]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-08T20:03:15.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:15 smithi083 ceph-mon[38695]: osdmap e15: 2 total, 2 up, 2 in 2024-05-08T20:03:16.022 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:15 smithi188 ceph-mon[48761]: osdmap e15: 2 total, 2 up, 2 in 2024-05-08T20:03:16.441 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-08T20:03:16.441 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-08T20:03:16.441 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10+0 records in 2024-05-08T20:03:16.441 INFO:teuthology.orchestra.run.smithi083.stderr:10+0 records out 2024-05-08T20:03:16.441 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130126 s, 806 MB/s 2024-05-08T20:03:16.441 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping successful for: 2024-05-08T20:03:17.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:16 smithi188 ceph-mon[48761]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:17.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:16 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:17.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:16 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:17.150 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:16 smithi083 ceph-mon[38695]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:17.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:16 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:17.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:16 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:17.170 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi083:vg_nvme/lv_2 2024-05-08T20:03:18.118 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:03:18.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:18 smithi083 ceph-mon[38695]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:19.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:18 smithi188 ceph-mon[48761]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:21.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:20 smithi188 ceph-mon[48761]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:21.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:20 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-08T20:03:21.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:20 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-08T20:03:21.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:20 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:03:21.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:20 smithi083 ceph-mon[38695]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:21.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:20 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-08T20:03:21.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:20 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-08T20:03:21.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:20 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:03:22.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:21 smithi188 ceph-mon[48761]: from='client.14284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:03:22.098 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:21 smithi083 ceph-mon[38695]: from='client.14284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:03:23.171 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:22 smithi083 ceph-mon[38695]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:23.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:22 smithi188 ceph-mon[48761]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:24.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:23 smithi188 ceph-mon[48761]: Detected new or changed devices on smithi083 2024-05-08T20:03:24.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:24.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:24.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:03:24.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:03:24.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:23 smithi188 ceph-mon[48761]: Adjusting osd_memory_target on smithi083 to 5937M 2024-05-08T20:03:24.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:24.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:03:24.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:03:24.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:24.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:03:24.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:03:24.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:23 smithi083 ceph-mon[38695]: Detected new or changed devices on smithi083 2024-05-08T20:03:24.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:24.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:24.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:03:24.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:03:24.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:23 smithi083 ceph-mon[38695]: Adjusting osd_memory_target on smithi083 to 5937M 2024-05-08T20:03:24.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:24.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:03:24.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:03:24.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:24.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:03:24.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:03:25.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:24 smithi083 ceph-mon[38695]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:25.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:24 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/949322982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ae0453a-718b-445f-ad00-a3696fec1c8b"}]: dispatch 2024-05-08T20:03:25.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:24 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/949322982' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4ae0453a-718b-445f-ad00-a3696fec1c8b"}]': finished 2024-05-08T20:03:25.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:24 smithi083 ceph-mon[38695]: osdmap e16: 3 total, 2 up, 3 in 2024-05-08T20:03:25.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-08T20:03:25.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:24 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2096718952' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-08T20:03:25.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:24 smithi188 ceph-mon[48761]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:25.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:24 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/949322982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ae0453a-718b-445f-ad00-a3696fec1c8b"}]: dispatch 2024-05-08T20:03:25.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:24 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/949322982' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4ae0453a-718b-445f-ad00-a3696fec1c8b"}]': finished 2024-05-08T20:03:25.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:24 smithi188 ceph-mon[48761]: osdmap e16: 3 total, 2 up, 3 in 2024-05-08T20:03:25.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:24 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-08T20:03:25.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:24 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/2096718952' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-08T20:03:26.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:26 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:26.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:26 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:26.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:26 smithi188 ceph-mon[48761]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:26.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:26 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:26.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:26 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:26.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:26 smithi083 ceph-mon[38695]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:29.030 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:28 smithi083 ceph-mon[38695]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:29.030 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:28 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:29.030 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:28 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:29.030 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:28 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:03:29.030 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:28 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:03:29.030 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:28 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:29.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:28 smithi188 ceph-mon[48761]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:29.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:28 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:29.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:28 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:29.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:28 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:03:29.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:28 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:03:29.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:28 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:31.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:30 smithi188 ceph-mon[48761]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:31.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:30 smithi083 ceph-mon[38695]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:32.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:31 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-08T20:03:32.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:31 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:03:32.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:31 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-08T20:03:32.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:31 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:03:33.130 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:32 smithi083 ceph-mon[38695]: Deploying daemon osd.2 on smithi083 2024-05-08T20:03:33.130 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:32 smithi083 ceph-mon[38695]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:33.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:32 smithi188 ceph-mon[48761]: Deploying daemon osd.2 on smithi083 2024-05-08T20:03:33.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:32 smithi188 ceph-mon[48761]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:35.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:34 smithi083 ceph-mon[38695]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:35.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:34 smithi188 ceph-mon[48761]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:37.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:36 smithi083 ceph-mon[38695]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:37.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:36 smithi188 ceph-mon[48761]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:38.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:37 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:03:38.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:37 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:38.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:37 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:38.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:37 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:03:38.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:37 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:38.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:37 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:39.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:39 smithi083 ceph-mon[38695]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:39.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:03:39.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:39 smithi188 ceph-mon[48761]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:39.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:03:41.247 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:41 smithi083 ceph-mon[38695]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:41.247 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:41 smithi083 ceph-mon[38695]: from='osd.2 [v2:172.21.15.83:6818/561864588,v1:172.21.15.83:6819/561864588]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-08T20:03:41.325 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 2 on host 'smithi083' 2024-05-08T20:03:41.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:41 smithi188 ceph-mon[48761]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:41.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:41 smithi188 ceph-mon[48761]: from='osd.2 [v2:172.21.15.83:6818/561864588,v1:172.21.15.83:6819/561864588]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-08T20:03:42.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:42 smithi083 ceph-mon[38695]: from='osd.2 [v2:172.21.15.83:6818/561864588,v1:172.21.15.83:6819/561864588]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-08T20:03:42.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:42 smithi083 ceph-mon[38695]: osdmap e17: 3 total, 2 up, 3 in 2024-05-08T20:03:42.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:42 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-08T20:03:42.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:42 smithi083 ceph-mon[38695]: from='osd.2 [v2:172.21.15.83:6818/561864588,v1:172.21.15.83:6819/561864588]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-05-08T20:03:42.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:42 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:42.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:42 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:42.310 DEBUG:teuthology.orchestra.run.smithi083:osd.2> sudo journalctl -f -n 0 -u ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.2.service 2024-05-08T20:03:42.313 INFO:tasks.cephadm:Deploying osd.3 on smithi083 with /dev/vg_nvme/lv_1... 2024-05-08T20:03:42.313 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-08T20:03:42.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:42 smithi188 ceph-mon[48761]: from='osd.2 [v2:172.21.15.83:6818/561864588,v1:172.21.15.83:6819/561864588]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-08T20:03:42.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:42 smithi188 ceph-mon[48761]: osdmap e17: 3 total, 2 up, 3 in 2024-05-08T20:03:42.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:42 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-08T20:03:42.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:42 smithi188 ceph-mon[48761]: from='osd.2 [v2:172.21.15.83:6818/561864588,v1:172.21.15.83:6819/561864588]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-05-08T20:03:42.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:42 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:42.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:42 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:42.342 INFO:journalctl@ceph.osd.2.smithi083.stdout:-- Logs begin at Wed 2024-05-08 19:47:49 UTC. -- 2024-05-08T20:03:42.682 INFO:journalctl@ceph.osd.2.smithi083.stdout:May 08 20:03:42 smithi083 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-2[79759]: 2024-05-08T20:03:42.499+0000 7ff183909700 -1 osd.2 0 waiting for initial osdmap 2024-05-08T20:03:42.683 INFO:journalctl@ceph.osd.2.smithi083.stdout:May 08 20:03:42 smithi083 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-2[79759]: 2024-05-08T20:03:42.502+0000 7ff17b6f6700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T20:03:43.134 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:03:43.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:43 smithi188 ceph-mon[48761]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:43.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:43 smithi188 ceph-mon[48761]: from='osd.2 [v2:172.21.15.83:6818/561864588,v1:172.21.15.83:6819/561864588]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-05-08T20:03:43.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:43 smithi188 ceph-mon[48761]: osdmap e18: 3 total, 2 up, 3 in 2024-05-08T20:03:43.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:43 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-08T20:03:43.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:43 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-08T20:03:43.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:43 smithi188 ceph-mon[48761]: from='osd.2 [v2:172.21.15.83:6818/561864588,v1:172.21.15.83:6819/561864588]' entity='osd.2' 2024-05-08T20:03:43.403 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:43 smithi083 ceph-mon[38695]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:43.404 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:43 smithi083 ceph-mon[38695]: from='osd.2 [v2:172.21.15.83:6818/561864588,v1:172.21.15.83:6819/561864588]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-05-08T20:03:43.404 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:43 smithi083 ceph-mon[38695]: osdmap e18: 3 total, 2 up, 3 in 2024-05-08T20:03:43.404 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:43 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-08T20:03:43.404 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:43 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-08T20:03:43.404 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:43 smithi083 ceph-mon[38695]: from='osd.2 [v2:172.21.15.83:6818/561864588,v1:172.21.15.83:6819/561864588]' entity='osd.2' 2024-05-08T20:03:44.133 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:44 smithi083 ceph-mon[38695]: purged_snaps scrub starts 2024-05-08T20:03:44.133 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:44 smithi083 ceph-mon[38695]: purged_snaps scrub ok 2024-05-08T20:03:44.133 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:44 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-08T20:03:44.133 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:44 smithi083 ceph-mon[38695]: osd.2 [v2:172.21.15.83:6818/561864588,v1:172.21.15.83:6819/561864588] boot 2024-05-08T20:03:44.133 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:44 smithi083 ceph-mon[38695]: osdmap e19: 3 total, 3 up, 3 in 2024-05-08T20:03:44.133 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:44 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-08T20:03:44.133 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:44 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-08T20:03:44.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:44 smithi188 ceph-mon[48761]: purged_snaps scrub starts 2024-05-08T20:03:44.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:44 smithi188 ceph-mon[48761]: purged_snaps scrub ok 2024-05-08T20:03:44.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:44 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-08T20:03:44.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:44 smithi188 ceph-mon[48761]: osd.2 [v2:172.21.15.83:6818/561864588,v1:172.21.15.83:6819/561864588] boot 2024-05-08T20:03:44.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:44 smithi188 ceph-mon[48761]: osdmap e19: 3 total, 3 up, 3 in 2024-05-08T20:03:44.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:44 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-08T20:03:44.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:44 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-08T20:03:45.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:45 smithi188 ceph-mon[48761]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:45.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:45 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-08T20:03:45.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:45 smithi188 ceph-mon[48761]: osdmap e20: 3 total, 3 up, 3 in 2024-05-08T20:03:45.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:45 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-08T20:03:45.383 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:45 smithi083 ceph-mon[38695]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-08T20:03:45.384 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:45 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-08T20:03:45.384 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:45 smithi083 ceph-mon[38695]: osdmap e20: 3 total, 3 up, 3 in 2024-05-08T20:03:45.384 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:45 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-08T20:03:46.178 INFO:journalctl@ceph.osd.0.smithi083.stdout:May 08 20:03:45 smithi083 sudo[81414]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-08T20:03:46.178 INFO:journalctl@ceph.osd.0.smithi083.stdout:May 08 20:03:45 smithi083 sudo[81414]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-08T20:03:46.178 INFO:journalctl@ceph.osd.0.smithi083.stdout:May 08 20:03:45 smithi083 sudo[81414]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-08T20:03:46.178 INFO:journalctl@ceph.osd.0.smithi083.stdout:May 08 20:03:45 smithi083 sudo[81414]: pam_unix(sudo:session): session closed for user root 2024-05-08T20:03:46.334 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-08T20:03:46.335 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-08T20:03:46.335 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10+0 records in 2024-05-08T20:03:46.335 INFO:teuthology.orchestra.run.smithi083.stderr:10+0 records out 2024-05-08T20:03:46.335 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109896 s, 954 MB/s 2024-05-08T20:03:46.335 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping successful for: 2024-05-08T20:03:46.432 INFO:journalctl@ceph.osd.0.smithi083.stdout:May 08 20:03:46 smithi083 sudo[81508]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-08T20:03:46.432 INFO:journalctl@ceph.osd.0.smithi083.stdout:May 08 20:03:46 smithi083 sudo[81508]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-08T20:03:46.432 INFO:journalctl@ceph.osd.0.smithi083.stdout:May 08 20:03:46 smithi083 sudo[81508]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-08T20:03:46.432 INFO:journalctl@ceph.osd.0.smithi083.stdout:May 08 20:03:46 smithi083 sudo[81508]: pam_unix(sudo:session): session closed for user root 2024-05-08T20:03:46.720 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:46 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-08T20:03:46.721 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:46 smithi083 ceph-mon[38695]: osdmap e21: 3 total, 3 up, 3 in 2024-05-08T20:03:46.721 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:46 smithi083 ceph-mon[38695]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:03:46.721 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:46 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:46.721 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:46 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:46.721 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:46 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:46.721 INFO:journalctl@ceph.osd.1.smithi083.stdout:May 08 20:03:46 smithi083 sudo[81702]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-08T20:03:46.721 INFO:journalctl@ceph.osd.1.smithi083.stdout:May 08 20:03:46 smithi083 sudo[81702]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-08T20:03:46.721 INFO:journalctl@ceph.osd.1.smithi083.stdout:May 08 20:03:46 smithi083 sudo[81702]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-08T20:03:46.722 INFO:journalctl@ceph.osd.1.smithi083.stdout:May 08 20:03:46 smithi083 sudo[81702]: pam_unix(sudo:session): session closed for user root 2024-05-08T20:03:46.832 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:46 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-08T20:03:46.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:46 smithi188 ceph-mon[48761]: osdmap e21: 3 total, 3 up, 3 in 2024-05-08T20:03:46.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:46 smithi188 ceph-mon[48761]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:03:46.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:46 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:46.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:46 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:46.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:46 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:46.978 INFO:journalctl@ceph.osd.1.smithi083.stdout:May 08 20:03:46 smithi083 sudo[81725]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-08T20:03:46.978 INFO:journalctl@ceph.osd.1.smithi083.stdout:May 08 20:03:46 smithi083 sudo[81725]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-08T20:03:46.978 INFO:journalctl@ceph.osd.1.smithi083.stdout:May 08 20:03:46 smithi083 sudo[81725]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-08T20:03:46.978 INFO:journalctl@ceph.osd.1.smithi083.stdout:May 08 20:03:46 smithi083 sudo[81725]: pam_unix(sudo:session): session closed for user root 2024-05-08T20:03:46.978 INFO:journalctl@ceph.osd.2.smithi083.stdout:May 08 20:03:46 smithi083 sudo[81735]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-08T20:03:46.978 INFO:journalctl@ceph.osd.2.smithi083.stdout:May 08 20:03:46 smithi083 sudo[81735]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-08T20:03:46.978 INFO:journalctl@ceph.osd.2.smithi083.stdout:May 08 20:03:46 smithi083 sudo[81735]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-08T20:03:46.979 INFO:journalctl@ceph.osd.2.smithi083.stdout:May 08 20:03:46 smithi083 sudo[81735]: pam_unix(sudo:session): session closed for user root 2024-05-08T20:03:47.202 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi083:vg_nvme/lv_1 2024-05-08T20:03:47.231 INFO:journalctl@ceph.osd.2.smithi083.stdout:May 08 20:03:47 smithi083 sudo[81748]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-08T20:03:47.231 INFO:journalctl@ceph.osd.2.smithi083.stdout:May 08 20:03:47 smithi083 sudo[81748]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-08T20:03:47.231 INFO:journalctl@ceph.osd.2.smithi083.stdout:May 08 20:03:47 smithi083 sudo[81748]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-08T20:03:47.231 INFO:journalctl@ceph.osd.2.smithi083.stdout:May 08 20:03:47 smithi083 sudo[81748]: pam_unix(sudo:session): session closed for user root 2024-05-08T20:03:47.484 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:47 smithi083 sudo[81814]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-08T20:03:47.484 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:47 smithi083 sudo[81814]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-08T20:03:47.484 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:47 smithi083 sudo[81814]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-08T20:03:47.484 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:47 smithi083 sudo[81814]: pam_unix(sudo:session): session closed for user root 2024-05-08T20:03:47.846 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:47 smithi083 sudo[81824]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-08T20:03:47.846 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:47 smithi083 sudo[81824]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-08T20:03:47.846 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:47 smithi083 sudo[81824]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-08T20:03:47.846 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:47 smithi083 sudo[81824]: pam_unix(sudo:session): session closed for user root 2024-05-08T20:03:47.846 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:47 smithi083 ceph-mon[38695]: osdmap e22: 3 total, 3 up, 3 in 2024-05-08T20:03:47.846 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:47 smithi083 ceph-mon[38695]: mgrmap e19: smithi083.fgcgxm(active, since 2m), standbys: smithi188.abfynx 2024-05-08T20:03:47.847 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:47 smithi083 ceph-mon[38695]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-08T20:03:48.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:47 smithi188 ceph-mon[48761]: osdmap e22: 3 total, 3 up, 3 in 2024-05-08T20:03:48.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:47 smithi188 ceph-mon[48761]: mgrmap e19: smithi083.fgcgxm(active, since 2m), standbys: smithi188.abfynx 2024-05-08T20:03:48.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:47 smithi188 ceph-mon[48761]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-08T20:03:48.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:47 smithi188 sudo[52550]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-08T20:03:48.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:47 smithi188 sudo[52550]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-08T20:03:48.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:47 smithi188 sudo[52550]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-08T20:03:48.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:47 smithi188 sudo[52550]: pam_unix(sudo:session): session closed for user root 2024-05-08T20:03:48.175 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:03:48.582 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:48 smithi188 sudo[52553]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-08T20:03:48.583 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:48 smithi188 sudo[52553]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-08T20:03:48.583 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:48 smithi188 sudo[52553]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-08T20:03:48.583 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:48 smithi188 sudo[52553]: pam_unix(sudo:session): session closed for user root 2024-05-08T20:03:49.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:48 smithi188 ceph-mon[48761]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-08T20:03:49.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:48 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi083"}]: dispatch 2024-05-08T20:03:49.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:48 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-05-08T20:03:49.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:48 smithi188 ceph-mon[48761]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-08T20:03:49.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:48 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi083"}]: dispatch 2024-05-08T20:03:49.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:48 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-05-08T20:03:49.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:48 smithi188 ceph-mon[48761]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:03:49.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:48 smithi188 ceph-mon[48761]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-08T20:03:49.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:48 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:49.169 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:48 smithi083 ceph-mon[38695]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-08T20:03:49.170 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:48 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi083"}]: dispatch 2024-05-08T20:03:49.170 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:48 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-05-08T20:03:49.170 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:48 smithi083 ceph-mon[38695]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-08T20:03:49.170 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:48 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi083"}]: dispatch 2024-05-08T20:03:49.170 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:48 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-05-08T20:03:49.170 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:48 smithi083 ceph-mon[38695]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:03:49.170 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:48 smithi083 ceph-mon[38695]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-08T20:03:49.170 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:48 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:51.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:50 smithi188 ceph-mon[48761]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:03:51.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:50 smithi083 ceph-mon[38695]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:03:51.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:51 smithi083 ceph-mon[38695]: from='client.14308 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:03:51.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:51 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-08T20:03:51.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:51 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-08T20:03:51.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:51 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:03:51.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:51 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:51.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:51 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:51.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:51 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:03:51.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:51 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:03:51.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:51 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:03:51.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:51 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:51.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:51 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:03:51.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:51 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:03:51.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:51 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:51.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:51 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:03:52.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:51 smithi188 ceph-mon[48761]: from='client.14308 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:03:52.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:51 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-08T20:03:52.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:51 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-08T20:03:52.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:51 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:03:52.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:51 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:52.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:51 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:52.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:51 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:03:52.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:51 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:03:52.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:51 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:03:52.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:51 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:52.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:51 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:03:52.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:51 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:03:52.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:51 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:52.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:51 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:03:52.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:52 smithi083 ceph-mon[38695]: Detected new or changed devices on smithi083 2024-05-08T20:03:52.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:52 smithi083 ceph-mon[38695]: Adjusting osd_memory_target on smithi083 to 3958M 2024-05-08T20:03:52.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:52 smithi083 ceph-mon[38695]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:03:53.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:52 smithi188 ceph-mon[48761]: Detected new or changed devices on smithi083 2024-05-08T20:03:53.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:52 smithi188 ceph-mon[48761]: Adjusting osd_memory_target on smithi083 to 3958M 2024-05-08T20:03:53.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:52 smithi188 ceph-mon[48761]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:03:54.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:53 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:03:54.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:53 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:03:55.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:54 smithi083 ceph-mon[38695]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:03:55.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:54 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/4268903423' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2085ec85-187a-4d69-a721-793cfed7d044"}]: dispatch 2024-05-08T20:03:55.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:54 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/4268903423' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2085ec85-187a-4d69-a721-793cfed7d044"}]': finished 2024-05-08T20:03:55.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:54 smithi083 ceph-mon[38695]: osdmap e23: 4 total, 3 up, 4 in 2024-05-08T20:03:55.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-08T20:03:55.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:54 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3706404838' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-08T20:03:55.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:54 smithi188 ceph-mon[48761]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:03:55.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:54 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/4268903423' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2085ec85-187a-4d69-a721-793cfed7d044"}]: dispatch 2024-05-08T20:03:55.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:54 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/4268903423' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2085ec85-187a-4d69-a721-793cfed7d044"}]': finished 2024-05-08T20:03:55.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:54 smithi188 ceph-mon[48761]: osdmap e23: 4 total, 3 up, 4 in 2024-05-08T20:03:55.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-08T20:03:55.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:54 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/3706404838' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-08T20:03:57.104 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:56 smithi083 ceph-mon[38695]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:03:57.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:56 smithi188 ceph-mon[48761]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:03:59.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:59 smithi188 ceph-mon[48761]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:03:59.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:59.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:59.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:03:59.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:03:59.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:03:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:59.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:59 smithi083 ceph-mon[38695]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:03:59.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:59.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:03:59.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:03:59.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:03:59.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:03:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:01.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:01 smithi083 ceph-mon[38695]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:04:01.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:01 smithi188 ceph-mon[48761]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:04:02.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-08T20:04:02.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:04:02.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-08T20:04:02.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:04:03.327 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:03 smithi083 ceph-mon[38695]: Deploying daemon osd.3 on smithi083 2024-05-08T20:04:03.327 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:03 smithi083 ceph-mon[38695]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:04:03.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:03 smithi188 ceph-mon[48761]: Deploying daemon osd.3 on smithi083 2024-05-08T20:04:03.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:03 smithi188 ceph-mon[48761]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:04:05.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:05 smithi188 ceph-mon[48761]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:04:05.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:05 smithi083 ceph-mon[38695]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:04:07.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:07 smithi188 ceph-mon[48761]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:04:07.365 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:07 smithi083 ceph-mon[38695]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:04:08.832 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:08.832 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:08.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:04:08.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:08 smithi188 ceph-mon[48761]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:04:08.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:08.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:08.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:04:08.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:08 smithi083 ceph-mon[38695]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:04:09.832 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:04:09.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:04:10.832 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:10 smithi188 ceph-mon[48761]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:04:10.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:10 smithi083 ceph-mon[38695]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:04:11.810 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:11 smithi083 ceph-mon[38695]: from='osd.3 [v2:172.21.15.83:6826/866184273,v1:172.21.15.83:6827/866184273]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-08T20:04:11.832 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:11 smithi188 ceph-mon[48761]: from='osd.3 [v2:172.21.15.83:6826/866184273,v1:172.21.15.83:6827/866184273]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-08T20:04:12.209 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 3 on host 'smithi083' 2024-05-08T20:04:12.832 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:12 smithi188 ceph-mon[48761]: from='osd.3 [v2:172.21.15.83:6826/866184273,v1:172.21.15.83:6827/866184273]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-08T20:04:12.832 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:12 smithi188 ceph-mon[48761]: osdmap e24: 4 total, 3 up, 4 in 2024-05-08T20:04:12.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:12 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-08T20:04:12.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:12 smithi188 ceph-mon[48761]: from='osd.3 [v2:172.21.15.83:6826/866184273,v1:172.21.15.83:6827/866184273]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-05-08T20:04:12.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:12 smithi188 ceph-mon[48761]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:04:12.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:12 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:12.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:12 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:12.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:12 smithi083 ceph-mon[38695]: from='osd.3 [v2:172.21.15.83:6826/866184273,v1:172.21.15.83:6827/866184273]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-08T20:04:12.934 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:12 smithi083 ceph-mon[38695]: osdmap e24: 4 total, 3 up, 4 in 2024-05-08T20:04:12.934 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:12 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-08T20:04:12.934 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:12 smithi083 ceph-mon[38695]: from='osd.3 [v2:172.21.15.83:6826/866184273,v1:172.21.15.83:6827/866184273]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-05-08T20:04:12.934 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:12 smithi083 ceph-mon[38695]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:04:12.934 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:12 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:12.934 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:12 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:12.979 DEBUG:teuthology.orchestra.run.smithi083:osd.3> sudo journalctl -f -n 0 -u ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.3.service 2024-05-08T20:04:13.021 INFO:tasks.cephadm:Deploying osd.4 on smithi188 with /dev/vg_nvme/lv_4... 2024-05-08T20:04:13.022 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-08T20:04:13.051 INFO:journalctl@ceph.osd.3.smithi083.stdout:-- Logs begin at Wed 2024-05-08 19:47:49 UTC. -- 2024-05-08T20:04:13.252 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi188/config 2024-05-08T20:04:13.808 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:13 smithi083 ceph-mon[38695]: from='osd.3 [v2:172.21.15.83:6826/866184273,v1:172.21.15.83:6827/866184273]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-05-08T20:04:13.808 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:13 smithi083 ceph-mon[38695]: osdmap e25: 4 total, 3 up, 4 in 2024-05-08T20:04:13.808 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:13 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-08T20:04:13.808 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:13 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-08T20:04:13.809 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:13 smithi083 ceph-mon[38695]: from='osd.3 [v2:172.21.15.83:6826/866184273,v1:172.21.15.83:6827/866184273]' entity='osd.3' 2024-05-08T20:04:13.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:13 smithi188 ceph-mon[48761]: from='osd.3 [v2:172.21.15.83:6826/866184273,v1:172.21.15.83:6827/866184273]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-05-08T20:04:13.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:13 smithi188 ceph-mon[48761]: osdmap e25: 4 total, 3 up, 4 in 2024-05-08T20:04:13.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:13 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-08T20:04:13.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:13 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-08T20:04:13.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:13 smithi188 ceph-mon[48761]: from='osd.3 [v2:172.21.15.83:6826/866184273,v1:172.21.15.83:6827/866184273]' entity='osd.3' 2024-05-08T20:04:14.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:14 smithi083 ceph-mon[38695]: purged_snaps scrub starts 2024-05-08T20:04:14.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:14 smithi083 ceph-mon[38695]: purged_snaps scrub ok 2024-05-08T20:04:14.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:14 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-08T20:04:14.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:14 smithi083 ceph-mon[38695]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:04:14.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:14 smithi083 ceph-mon[38695]: osd.3 [v2:172.21.15.83:6826/866184273,v1:172.21.15.83:6827/866184273] boot 2024-05-08T20:04:14.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:14 smithi083 ceph-mon[38695]: osdmap e26: 4 total, 4 up, 4 in 2024-05-08T20:04:14.683 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:14 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-08T20:04:14.803 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:14 smithi188 ceph-mon[48761]: purged_snaps scrub starts 2024-05-08T20:04:14.803 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:14 smithi188 ceph-mon[48761]: purged_snaps scrub ok 2024-05-08T20:04:14.803 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:14 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-08T20:04:14.803 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:14 smithi188 ceph-mon[48761]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-08T20:04:14.803 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:14 smithi188 ceph-mon[48761]: osd.3 [v2:172.21.15.83:6826/866184273,v1:172.21.15.83:6827/866184273] boot 2024-05-08T20:04:14.804 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:14 smithi188 ceph-mon[48761]: osdmap e26: 4 total, 4 up, 4 in 2024-05-08T20:04:14.804 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:14 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-08T20:04:15.398 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-08T20:04:15.398 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-08T20:04:15.399 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-05-08T20:04:15.399 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-05-08T20:04:15.399 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111991 s, 936 MB/s 2024-05-08T20:04:15.399 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-05-08T20:04:16.127 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_4 2024-05-08T20:04:16.354 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi188/config 2024-05-08T20:04:16.615 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:16 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:16.615 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:16 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:16.615 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:16 smithi188 ceph-mon[48761]: osdmap e27: 4 total, 4 up, 4 in 2024-05-08T20:04:16.615 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:16 smithi188 ceph-mon[48761]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:16.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:16 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:16.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:16 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:16.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:16 smithi083 ceph-mon[38695]: osdmap e27: 4 total, 4 up, 4 in 2024-05-08T20:04:16.682 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:16 smithi083 ceph-mon[38695]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:18.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:18 smithi083 ceph-mon[38695]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:18.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:18 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-08T20:04:18.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:18 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-08T20:04:18.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:18 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:04:19.160 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:18 smithi188 ceph-mon[48761]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:19.160 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:18 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-08T20:04:19.160 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:18 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-08T20:04:19.161 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:18 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:04:20.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:20 smithi188 ceph-mon[48761]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:04:20.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:20 smithi188 ceph-mon[48761]: Detected new or changed devices on smithi083 2024-05-08T20:04:20.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:20 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:20.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:20 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:20.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:20 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:04:20.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:20 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:04:20.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:20 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:04:20.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:20 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:04:20.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:20 smithi188 ceph-mon[48761]: Adjusting osd_memory_target on smithi083 to 2968M 2024-05-08T20:04:20.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:20 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:20.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:20 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:04:20.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:20 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:04:20.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:20 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:20.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:20 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:04:20.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:20 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:04:20.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:20 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:04:20.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:20 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:20.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:20 smithi083 ceph-mon[38695]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:04:20.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:20 smithi083 ceph-mon[38695]: Detected new or changed devices on smithi083 2024-05-08T20:04:20.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:20 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:20.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:20 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:20.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:20 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:04:20.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:20 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:04:20.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:20 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:04:20.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:20 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:04:20.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:20 smithi083 ceph-mon[38695]: Adjusting osd_memory_target on smithi083 to 2968M 2024-05-08T20:04:20.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:20 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:20.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:20 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:04:20.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:20 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:04:20.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:20 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:20.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:20 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:04:20.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:20 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:04:20.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:20 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:04:20.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:20 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:21.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:21 smithi188 ceph-mon[48761]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:21.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:21 smithi188 ceph-mon[48761]: from='client.? 172.21.15.188:0/1511929927' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b92dad8b-5bd0-4cd1-ba8d-2920927f132e"}]: dispatch 2024-05-08T20:04:21.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:21 smithi188 ceph-mon[48761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b92dad8b-5bd0-4cd1-ba8d-2920927f132e"}]: dispatch 2024-05-08T20:04:21.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:21 smithi188 ceph-mon[48761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b92dad8b-5bd0-4cd1-ba8d-2920927f132e"}]': finished 2024-05-08T20:04:21.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:21 smithi188 ceph-mon[48761]: osdmap e28: 5 total, 4 up, 5 in 2024-05-08T20:04:21.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:21 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-08T20:04:21.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:21 smithi083 ceph-mon[38695]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:21.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:21 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/1511929927' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b92dad8b-5bd0-4cd1-ba8d-2920927f132e"}]: dispatch 2024-05-08T20:04:21.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:21 smithi083 ceph-mon[38695]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b92dad8b-5bd0-4cd1-ba8d-2920927f132e"}]: dispatch 2024-05-08T20:04:21.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:21 smithi083 ceph-mon[38695]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b92dad8b-5bd0-4cd1-ba8d-2920927f132e"}]': finished 2024-05-08T20:04:21.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:21 smithi083 ceph-mon[38695]: osdmap e28: 5 total, 4 up, 5 in 2024-05-08T20:04:21.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:21 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-08T20:04:22.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:22 smithi188 ceph-mon[48761]: from='client.? 172.21.15.188:0/1295080522' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-08T20:04:22.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:22 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/1295080522' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-08T20:04:23.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:23 smithi188 ceph-mon[48761]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:23.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:23 smithi083 ceph-mon[38695]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:24.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:24 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:04:24.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:04:25.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:25 smithi188 ceph-mon[48761]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:25.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:25 smithi083 ceph-mon[38695]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:27.185 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:27 smithi188 ceph-mon[48761]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:27.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:27 smithi083 ceph-mon[38695]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:28.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:28 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-08T20:04:28.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:28 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:04:28.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:28 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-08T20:04:28.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:28 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:04:29.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:29 smithi188 ceph-mon[48761]: Deploying daemon osd.4 on smithi188 2024-05-08T20:04:29.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:29 smithi188 ceph-mon[48761]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:29.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:29 smithi083 ceph-mon[38695]: Deploying daemon osd.4 on smithi188 2024-05-08T20:04:29.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:29 smithi083 ceph-mon[38695]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:31.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:31 smithi188 ceph-mon[48761]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:31.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:31 smithi083 ceph-mon[38695]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:33.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:33 smithi188 ceph-mon[48761]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:33.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:33 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:33.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:33 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:33.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:33 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:04:33.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:33 smithi083 ceph-mon[38695]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:33.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:33 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:33.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:33 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:33.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:33 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:04:35.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:35 smithi188 ceph-mon[48761]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:35.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:35 smithi083 ceph-mon[38695]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:36.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:36 smithi188 ceph-mon[48761]: from='osd.4 [v2:172.21.15.188:6800/422247785,v1:172.21.15.188:6801/422247785]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-08T20:04:36.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:36 smithi188 ceph-mon[48761]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-08T20:04:36.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:36 smithi188 ceph-mon[48761]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:36.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:36 smithi083 ceph-mon[38695]: from='osd.4 [v2:172.21.15.188:6800/422247785,v1:172.21.15.188:6801/422247785]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-08T20:04:36.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:36 smithi083 ceph-mon[38695]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-08T20:04:36.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:36 smithi083 ceph-mon[38695]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:36.622 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 4 on host 'smithi188' 2024-05-08T20:04:37.265 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:37 smithi188 ceph-mon[48761]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-08T20:04:37.265 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:37 smithi188 ceph-mon[48761]: osdmap e29: 5 total, 4 up, 5 in 2024-05-08T20:04:37.265 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:37 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-08T20:04:37.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:37 smithi188 ceph-mon[48761]: from='osd.4 [v2:172.21.15.188:6800/422247785,v1:172.21.15.188:6801/422247785]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-05-08T20:04:37.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:37 smithi188 ceph-mon[48761]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-05-08T20:04:37.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:37 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:37.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:37 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:37.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:37 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:37.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:37 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:37.267 DEBUG:teuthology.orchestra.run.smithi188:osd.4> sudo journalctl -f -n 0 -u ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.4.service 2024-05-08T20:04:37.270 INFO:tasks.cephadm:Deploying osd.5 on smithi188 with /dev/vg_nvme/lv_3... 2024-05-08T20:04:37.270 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-08T20:04:37.303 INFO:journalctl@ceph.osd.4.smithi188.stdout:-- Logs begin at Wed 2024-05-08 19:48:17 UTC. -- 2024-05-08T20:04:37.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:37 smithi083 ceph-mon[38695]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-08T20:04:37.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:37 smithi083 ceph-mon[38695]: osdmap e29: 5 total, 4 up, 5 in 2024-05-08T20:04:37.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:37 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-08T20:04:37.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:37 smithi083 ceph-mon[38695]: from='osd.4 [v2:172.21.15.188:6800/422247785,v1:172.21.15.188:6801/422247785]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-05-08T20:04:37.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:37 smithi083 ceph-mon[38695]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-05-08T20:04:37.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:37 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:37.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:37 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:37.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:37 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:37.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:37 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:38.066 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi188/config 2024-05-08T20:04:38.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:38 smithi083 ceph-mon[38695]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-05-08T20:04:38.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:38 smithi083 ceph-mon[38695]: osdmap e30: 5 total, 4 up, 5 in 2024-05-08T20:04:38.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-08T20:04:38.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-08T20:04:38.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:38 smithi083 ceph-mon[38695]: osd.4 [v2:172.21.15.188:6800/422247785,v1:172.21.15.188:6801/422247785] boot 2024-05-08T20:04:38.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:38 smithi083 ceph-mon[38695]: osdmap e31: 5 total, 5 up, 5 in 2024-05-08T20:04:38.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-08T20:04:38.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:38 smithi083 ceph-mon[38695]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:38.506 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:38 smithi188 ceph-mon[48761]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-05-08T20:04:38.507 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:38 smithi188 ceph-mon[48761]: osdmap e30: 5 total, 4 up, 5 in 2024-05-08T20:04:38.507 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-08T20:04:38.507 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-08T20:04:38.507 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:38 smithi188 ceph-mon[48761]: osd.4 [v2:172.21.15.188:6800/422247785,v1:172.21.15.188:6801/422247785] boot 2024-05-08T20:04:38.507 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:38 smithi188 ceph-mon[48761]: osdmap e31: 5 total, 5 up, 5 in 2024-05-08T20:04:38.507 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-08T20:04:38.507 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:38 smithi188 ceph-mon[48761]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-08T20:04:39.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:39 smithi083 ceph-mon[38695]: purged_snaps scrub starts 2024-05-08T20:04:39.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:39 smithi083 ceph-mon[38695]: purged_snaps scrub ok 2024-05-08T20:04:39.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:39 smithi083 ceph-mon[38695]: osdmap e32: 5 total, 5 up, 5 in 2024-05-08T20:04:39.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:04:39.582 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:39 smithi188 ceph-mon[48761]: purged_snaps scrub starts 2024-05-08T20:04:39.583 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:39 smithi188 ceph-mon[48761]: purged_snaps scrub ok 2024-05-08T20:04:39.583 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:39 smithi188 ceph-mon[48761]: osdmap e32: 5 total, 5 up, 5 in 2024-05-08T20:04:39.583 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:04:40.832 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:40 smithi188 ceph-mon[48761]: osdmap e33: 5 total, 5 up, 5 in 2024-05-08T20:04:40.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:40 smithi188 ceph-mon[48761]: pgmap v119: 1 pgs: 1 active+recovering+undersized+degraded+remapped; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 2/6 objects misplaced (33.333%) 2024-05-08T20:04:41.142 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-08T20:04:41.142 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-08T20:04:41.142 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-05-08T20:04:41.142 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-05-08T20:04:41.142 INFO:teuthology.orchestra.run.smithi188.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0122002 s, 859 MB/s 2024-05-08T20:04:41.142 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-05-08T20:04:41.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:40 smithi083 ceph-mon[38695]: osdmap e33: 5 total, 5 up, 5 in 2024-05-08T20:04:41.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:40 smithi083 ceph-mon[38695]: pgmap v119: 1 pgs: 1 active+recovering+undersized+degraded+remapped; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 2/6 objects misplaced (33.333%) 2024-05-08T20:04:41.945 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:41 smithi188 ceph-mon[48761]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-05-08T20:04:41.945 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:41 smithi188 ceph-mon[48761]: osdmap e34: 5 total, 5 up, 5 in 2024-05-08T20:04:41.945 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:41 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:41.945 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:41 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:41.945 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:41 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:04:41.945 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:41 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:41.945 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:41 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:04:41.945 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:41 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:04:41.945 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:41 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:41.946 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:41 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:04:41.946 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:41 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:04:41.946 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:41 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:04:41.946 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:41 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:41.963 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_3 2024-05-08T20:04:42.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:41 smithi083 ceph-mon[38695]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-05-08T20:04:42.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:41 smithi083 ceph-mon[38695]: osdmap e34: 5 total, 5 up, 5 in 2024-05-08T20:04:42.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:41 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:42.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:41 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:42.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:41 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:04:42.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:41 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:42.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:41 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:04:42.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:41 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:04:42.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:41 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:42.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:41 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:04:42.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:41 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:04:42.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:41 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:04:42.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:41 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:04:42.184 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi188/config 2024-05-08T20:04:43.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:42 smithi188 ceph-mon[48761]: Detected new or changed devices on smithi188 2024-05-08T20:04:43.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:42 smithi188 ceph-mon[48761]: Adjusting osd_memory_target on smithi188 to 14946M 2024-05-08T20:04:43.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:42 smithi188 ceph-mon[48761]: osdmap e35: 5 total, 5 up, 5 in 2024-05-08T20:04:43.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:42 smithi188 ceph-mon[48761]: pgmap v122: 1 pgs: 1 active+recovering+undersized+degraded+remapped; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 2/6 objects misplaced (33.333%) 2024-05-08T20:04:43.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:42 smithi083 ceph-mon[38695]: Detected new or changed devices on smithi188 2024-05-08T20:04:43.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:42 smithi083 ceph-mon[38695]: Adjusting osd_memory_target on smithi188 to 14946M 2024-05-08T20:04:43.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:42 smithi083 ceph-mon[38695]: osdmap e35: 5 total, 5 up, 5 in 2024-05-08T20:04:43.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:42 smithi083 ceph-mon[38695]: pgmap v122: 1 pgs: 1 active+recovering+undersized+degraded+remapped; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 2/6 objects misplaced (33.333%) 2024-05-08T20:04:44.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:44 smithi083 ceph-mon[38695]: pgmap v123: 1 pgs: 1 active+recovering+undersized+degraded+remapped; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 2/6 objects misplaced (33.333%) 2024-05-08T20:04:44.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:44 smithi083 ceph-mon[38695]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:04:44.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:44 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-08T20:04:44.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:44 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-08T20:04:44.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:44 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:04:45.306 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:44 smithi188 ceph-mon[48761]: pgmap v123: 1 pgs: 1 active+recovering+undersized+degraded+remapped; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 2/6 objects misplaced (33.333%) 2024-05-08T20:04:45.306 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:44 smithi188 ceph-mon[48761]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:04:45.306 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:44 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-08T20:04:45.306 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:44 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-08T20:04:45.306 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:44 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:04:46.167 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:45 smithi188 ceph-mon[48761]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-05-08T20:04:46.167 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:45 smithi188 ceph-mon[48761]: Cluster is now healthy 2024-05-08T20:04:46.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:45 smithi083 ceph-mon[38695]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-05-08T20:04:46.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:45 smithi083 ceph-mon[38695]: Cluster is now healthy 2024-05-08T20:04:47.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:46 smithi083 ceph-mon[38695]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-05-08T20:04:47.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:46 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/1877429308' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aeeda050-668f-47a2-90d0-32e1aa35bfcb"}]: dispatch 2024-05-08T20:04:47.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:46 smithi083 ceph-mon[38695]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aeeda050-668f-47a2-90d0-32e1aa35bfcb"}]: dispatch 2024-05-08T20:04:47.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:46 smithi083 ceph-mon[38695]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aeeda050-668f-47a2-90d0-32e1aa35bfcb"}]': finished 2024-05-08T20:04:47.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:46 smithi083 ceph-mon[38695]: osdmap e36: 6 total, 5 up, 6 in 2024-05-08T20:04:47.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:46 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-08T20:04:47.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:46 smithi188 ceph-mon[48761]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-05-08T20:04:47.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:46 smithi188 ceph-mon[48761]: from='client.? 172.21.15.188:0/1877429308' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aeeda050-668f-47a2-90d0-32e1aa35bfcb"}]: dispatch 2024-05-08T20:04:47.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:46 smithi188 ceph-mon[48761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aeeda050-668f-47a2-90d0-32e1aa35bfcb"}]: dispatch 2024-05-08T20:04:47.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:46 smithi188 ceph-mon[48761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aeeda050-668f-47a2-90d0-32e1aa35bfcb"}]': finished 2024-05-08T20:04:47.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:46 smithi188 ceph-mon[48761]: osdmap e36: 6 total, 5 up, 6 in 2024-05-08T20:04:47.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:46 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-08T20:04:48.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:47 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/3804072319' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-08T20:04:48.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:47 smithi188 ceph-mon[48761]: from='client.? 172.21.15.188:0/3804072319' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-08T20:04:49.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:48 smithi083 ceph-mon[38695]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-05-08T20:04:49.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:48 smithi188 ceph-mon[48761]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-05-08T20:04:51.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:50 smithi083 ceph-mon[38695]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-08T20:04:51.185 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:50 smithi188 ceph-mon[48761]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-08T20:04:53.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:52 smithi083 ceph-mon[38695]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-08T20:04:53.250 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:52 smithi188 ceph-mon[48761]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-08T20:04:54.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:53 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-08T20:04:54.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:53 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:04:54.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:53 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:04:54.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:53 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-08T20:04:54.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:53 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:04:54.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:53 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:04:55.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:54 smithi188 ceph-mon[48761]: Deploying daemon osd.5 on smithi188 2024-05-08T20:04:55.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:54 smithi188 ceph-mon[48761]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-08T20:04:55.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:54 smithi083 ceph-mon[38695]: Deploying daemon osd.5 on smithi188 2024-05-08T20:04:55.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:54 smithi083 ceph-mon[38695]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-08T20:04:57.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:56 smithi083 ceph-mon[38695]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-08T20:04:57.275 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:56 smithi188 ceph-mon[48761]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-08T20:04:59.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:04:58 smithi083 ceph-mon[38695]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-08T20:04:59.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:04:58 smithi188 ceph-mon[48761]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-08T20:05:00.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:00 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:00.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:00 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:00.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:00 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:05:00.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:00 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:00.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:00 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:00.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:00 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:05:01.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:01 smithi188 ceph-mon[48761]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-08T20:05:01.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:01 smithi083 ceph-mon[38695]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-08T20:05:02.800 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 5 on host 'smithi188' 2024-05-08T20:05:03.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:03 smithi188 ceph-mon[48761]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-08T20:05:03.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:03 smithi188 ceph-mon[48761]: from='osd.5 [v2:172.21.15.188:6808/841652631,v1:172.21.15.188:6809/841652631]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-08T20:05:03.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:03 smithi188 ceph-mon[48761]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-08T20:05:03.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:03 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:03.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:03 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:03.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:03 smithi083 ceph-mon[38695]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-08T20:05:03.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:03 smithi083 ceph-mon[38695]: from='osd.5 [v2:172.21.15.188:6808/841652631,v1:172.21.15.188:6809/841652631]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-08T20:05:03.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:03 smithi083 ceph-mon[38695]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-08T20:05:03.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:03 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:03.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:03 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:03.829 DEBUG:teuthology.orchestra.run.smithi188:osd.5> sudo journalctl -f -n 0 -u ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.5.service 2024-05-08T20:05:03.832 INFO:tasks.cephadm:Deploying osd.6 on smithi188 with /dev/vg_nvme/lv_2... 2024-05-08T20:05:03.832 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-08T20:05:03.866 INFO:journalctl@ceph.osd.5.smithi188.stdout:-- Logs begin at Wed 2024-05-08 19:48:17 UTC. -- 2024-05-08T20:05:04.098 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi188/config 2024-05-08T20:05:04.324 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:05:04 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:05:04.082+0000 7f61697c4700 -1 osd.5 0 waiting for initial osdmap 2024-05-08T20:05:04.324 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:05:04 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:05:04.088+0000 7f61655b9700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T20:05:04.324 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:04 smithi188 ceph-mon[48761]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-08T20:05:04.324 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:04 smithi188 ceph-mon[48761]: osdmap e37: 6 total, 5 up, 6 in 2024-05-08T20:05:04.324 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:04 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-08T20:05:04.324 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:04 smithi188 ceph-mon[48761]: from='osd.5 [v2:172.21.15.188:6808/841652631,v1:172.21.15.188:6809/841652631]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-05-08T20:05:04.324 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:04 smithi188 ceph-mon[48761]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-05-08T20:05:04.324 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:04 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:04.324 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:04 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:04.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:04 smithi083 ceph-mon[38695]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-08T20:05:04.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:04 smithi083 ceph-mon[38695]: osdmap e37: 6 total, 5 up, 6 in 2024-05-08T20:05:04.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:04 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-08T20:05:04.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:04 smithi083 ceph-mon[38695]: from='osd.5 [v2:172.21.15.188:6808/841652631,v1:172.21.15.188:6809/841652631]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-05-08T20:05:04.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:04 smithi083 ceph-mon[38695]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-05-08T20:05:04.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:04 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:04.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:04 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:05.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:05 smithi083 ceph-mon[38695]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-08T20:05:05.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:05 smithi083 ceph-mon[38695]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-05-08T20:05:05.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:05 smithi083 ceph-mon[38695]: osdmap e38: 6 total, 5 up, 6 in 2024-05-08T20:05:05.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:05 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-08T20:05:05.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:05 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-08T20:05:05.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:05 smithi083 ceph-mon[38695]: osd.5 [v2:172.21.15.188:6808/841652631,v1:172.21.15.188:6809/841652631] boot 2024-05-08T20:05:05.324 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:05 smithi188 ceph-mon[48761]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-08T20:05:05.325 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:05 smithi188 ceph-mon[48761]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-05-08T20:05:05.325 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:05 smithi188 ceph-mon[48761]: osdmap e38: 6 total, 5 up, 6 in 2024-05-08T20:05:05.325 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:05 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-08T20:05:05.325 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:05 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-08T20:05:05.325 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:05 smithi188 ceph-mon[48761]: osd.5 [v2:172.21.15.188:6808/841652631,v1:172.21.15.188:6809/841652631] boot 2024-05-08T20:05:06.223 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-08T20:05:06.223 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-08T20:05:06.223 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-05-08T20:05:06.223 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-05-08T20:05:06.223 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110554 s, 948 MB/s 2024-05-08T20:05:06.223 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-05-08T20:05:06.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:06 smithi188 ceph-mon[48761]: purged_snaps scrub starts 2024-05-08T20:05:06.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:06 smithi188 ceph-mon[48761]: purged_snaps scrub ok 2024-05-08T20:05:06.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:06 smithi188 ceph-mon[48761]: osdmap e39: 6 total, 6 up, 6 in 2024-05-08T20:05:06.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:06 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-08T20:05:06.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:06 smithi188 ceph-mon[48761]: osdmap e40: 6 total, 6 up, 6 in 2024-05-08T20:05:06.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:06 smithi083 ceph-mon[38695]: purged_snaps scrub starts 2024-05-08T20:05:06.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:06 smithi083 ceph-mon[38695]: purged_snaps scrub ok 2024-05-08T20:05:06.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:06 smithi083 ceph-mon[38695]: osdmap e39: 6 total, 6 up, 6 in 2024-05-08T20:05:06.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:06 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-08T20:05:06.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:06 smithi083 ceph-mon[38695]: osdmap e40: 6 total, 6 up, 6 in 2024-05-08T20:05:07.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:07 smithi188 ceph-mon[48761]: pgmap v138: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-05-08T20:05:07.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:07 smithi188 ceph-mon[48761]: osdmap e41: 6 total, 6 up, 6 in 2024-05-08T20:05:07.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:07 smithi083 ceph-mon[38695]: pgmap v138: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-05-08T20:05:07.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:07 smithi083 ceph-mon[38695]: osdmap e41: 6 total, 6 up, 6 in 2024-05-08T20:05:08.085 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_2 2024-05-08T20:05:08.322 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi188/config 2024-05-08T20:05:08.920 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:08 smithi188 ceph-mon[48761]: osdmap e42: 6 total, 6 up, 6 in 2024-05-08T20:05:08.920 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:08 smithi188 ceph-mon[48761]: pgmap v142: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-05-08T20:05:09.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:08 smithi083 ceph-mon[38695]: osdmap e42: 6 total, 6 up, 6 in 2024-05-08T20:05:09.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:08 smithi083 ceph-mon[38695]: pgmap v142: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-05-08T20:05:10.069 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:09 smithi188 ceph-mon[48761]: osdmap e43: 6 total, 6 up, 6 in 2024-05-08T20:05:10.069 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:05:10.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:09 smithi083 ceph-mon[38695]: osdmap e43: 6 total, 6 up, 6 in 2024-05-08T20:05:10.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:05:11.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:10 smithi188 ceph-mon[48761]: pgmap v144: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-05-08T20:05:11.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:10 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:11.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:10 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:11.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:10 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:05:11.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:10 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:05:11.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:10 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:11.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:10 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:05:11.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:10 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:05:11.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:10 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:11.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:10 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:05:11.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:10 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:05:11.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:10 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:05:11.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:10 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:11.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:10 smithi083 ceph-mon[38695]: pgmap v144: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-05-08T20:05:11.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:10 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:11.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:10 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:11.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:10 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:05:11.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:10 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:05:11.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:10 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:11.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:10 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:05:11.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:10 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:05:11.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:10 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:11.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:10 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:05:11.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:10 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:05:11.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:10 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:05:11.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:10 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:12.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:11 smithi188 ceph-mon[48761]: Detected new or changed devices on smithi188 2024-05-08T20:05:12.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:11 smithi188 ceph-mon[48761]: Adjusting osd_memory_target on smithi188 to 7473M 2024-05-08T20:05:12.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:11 smithi188 ceph-mon[48761]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:05:12.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:11 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-08T20:05:12.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:11 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-08T20:05:12.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:11 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:05:12.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:11 smithi083 ceph-mon[38695]: Detected new or changed devices on smithi188 2024-05-08T20:05:12.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:11 smithi083 ceph-mon[38695]: Adjusting osd_memory_target on smithi188 to 7473M 2024-05-08T20:05:12.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:11 smithi083 ceph-mon[38695]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:05:12.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:11 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-08T20:05:12.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:11 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-08T20:05:12.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:11 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:05:13.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:12 smithi188 ceph-mon[48761]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-05-08T20:05:13.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:12 smithi083 ceph-mon[38695]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-05-08T20:05:14.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:13 smithi188 ceph-mon[48761]: from='client.? 172.21.15.188:0/600516737' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d3720c87-ec2b-490a-937b-a29833f0267c"}]: dispatch 2024-05-08T20:05:14.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:13 smithi188 ceph-mon[48761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d3720c87-ec2b-490a-937b-a29833f0267c"}]: dispatch 2024-05-08T20:05:14.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:13 smithi188 ceph-mon[48761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d3720c87-ec2b-490a-937b-a29833f0267c"}]': finished 2024-05-08T20:05:14.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:13 smithi188 ceph-mon[48761]: osdmap e44: 7 total, 6 up, 7 in 2024-05-08T20:05:14.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:13 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-08T20:05:14.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:13 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/600516737' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d3720c87-ec2b-490a-937b-a29833f0267c"}]: dispatch 2024-05-08T20:05:14.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:13 smithi083 ceph-mon[38695]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d3720c87-ec2b-490a-937b-a29833f0267c"}]: dispatch 2024-05-08T20:05:14.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:13 smithi083 ceph-mon[38695]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d3720c87-ec2b-490a-937b-a29833f0267c"}]': finished 2024-05-08T20:05:14.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:13 smithi083 ceph-mon[38695]: osdmap e44: 7 total, 6 up, 7 in 2024-05-08T20:05:14.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:13 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-08T20:05:15.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:14 smithi188 ceph-mon[48761]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-05-08T20:05:15.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:14 smithi188 ceph-mon[48761]: from='client.? 172.21.15.188:0/2196795574' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-08T20:05:15.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:14 smithi083 ceph-mon[38695]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-05-08T20:05:15.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:14 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/2196795574' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-08T20:05:17.169 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:16 smithi188 ceph-mon[48761]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-08T20:05:17.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:16 smithi083 ceph-mon[38695]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-08T20:05:18.982 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:18 smithi188 ceph-mon[48761]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-05-08T20:05:19.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:18 smithi083 ceph-mon[38695]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-05-08T20:05:20.929 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:20 smithi188 ceph-mon[48761]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-08T20:05:20.929 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:20 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-08T20:05:20.929 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:20 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:05:21.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:20 smithi083 ceph-mon[38695]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-08T20:05:21.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:20 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-08T20:05:21.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:20 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:05:21.971 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:21 smithi188 ceph-mon[48761]: Deploying daemon osd.6 on smithi188 2024-05-08T20:05:22.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:21 smithi083 ceph-mon[38695]: Deploying daemon osd.6 on smithi188 2024-05-08T20:05:22.912 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:22 smithi188 ceph-mon[48761]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-05-08T20:05:23.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:22 smithi083 ceph-mon[38695]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-05-08T20:05:24.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:05:24.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:05:25.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:24 smithi083 ceph-mon[38695]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-05-08T20:05:25.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:24 smithi188 ceph-mon[48761]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-05-08T20:05:27.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:26 smithi188 ceph-mon[48761]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-05-08T20:05:27.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:26 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:27.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:26 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:27.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:26 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:05:27.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:26 smithi083 ceph-mon[38695]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-05-08T20:05:27.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:26 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:27.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:26 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:27.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:26 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:05:29.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:28 smithi083 ceph-mon[38695]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-05-08T20:05:29.256 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:28 smithi188 ceph-mon[48761]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-05-08T20:05:29.613 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 6 on host 'smithi188' 2024-05-08T20:05:30.094 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:29 smithi083 ceph-mon[38695]: from='osd.6 [v2:172.21.15.188:6816/118382910,v1:172.21.15.188:6817/118382910]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-08T20:05:30.094 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:29 smithi083 ceph-mon[38695]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-08T20:05:30.094 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:29 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:30.095 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:29 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:30.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:29 smithi188 ceph-mon[48761]: from='osd.6 [v2:172.21.15.188:6816/118382910,v1:172.21.15.188:6817/118382910]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-08T20:05:30.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:29 smithi188 ceph-mon[48761]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-08T20:05:30.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:29 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:30.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:29 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:30.300 DEBUG:teuthology.orchestra.run.smithi188:osd.6> sudo journalctl -f -n 0 -u ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.6.service 2024-05-08T20:05:30.302 INFO:tasks.cephadm:Deploying osd.7 on smithi188 with /dev/vg_nvme/lv_1... 2024-05-08T20:05:30.303 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-08T20:05:30.332 INFO:journalctl@ceph.osd.6.smithi188.stdout:-- Logs begin at Wed 2024-05-08 19:48:17 UTC. -- 2024-05-08T20:05:30.564 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi188/config 2024-05-08T20:05:31.063 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:30 smithi188 ceph-mon[48761]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-05-08T20:05:31.063 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:30 smithi188 ceph-mon[48761]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-08T20:05:31.063 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:30 smithi188 ceph-mon[48761]: osdmap e45: 7 total, 6 up, 7 in 2024-05-08T20:05:31.063 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:30 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-08T20:05:31.063 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:30 smithi188 ceph-mon[48761]: from='osd.6 [v2:172.21.15.188:6816/118382910,v1:172.21.15.188:6817/118382910]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-05-08T20:05:31.063 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:30 smithi188 ceph-mon[48761]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-05-08T20:05:31.063 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:30 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:31.063 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:30 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:31.063 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:05:30 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:05:30.862+0000 7fd0bd188700 -1 osd.6 0 waiting for initial osdmap 2024-05-08T20:05:31.064 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:05:30 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:05:30.868+0000 7fd0b8f7d700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T20:05:31.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:30 smithi083 ceph-mon[38695]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-05-08T20:05:31.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:30 smithi083 ceph-mon[38695]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-08T20:05:31.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:30 smithi083 ceph-mon[38695]: osdmap e45: 7 total, 6 up, 7 in 2024-05-08T20:05:31.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:30 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-08T20:05:31.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:30 smithi083 ceph-mon[38695]: from='osd.6 [v2:172.21.15.188:6816/118382910,v1:172.21.15.188:6817/118382910]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-05-08T20:05:31.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:30 smithi083 ceph-mon[38695]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-05-08T20:05:31.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:30 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:31.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:30 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:31.953 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:31 smithi188 ceph-mon[48761]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-05-08T20:05:31.953 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:31 smithi188 ceph-mon[48761]: osdmap e46: 7 total, 6 up, 7 in 2024-05-08T20:05:31.953 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:31 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-08T20:05:32.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:31 smithi083 ceph-mon[38695]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-05-08T20:05:32.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:31 smithi083 ceph-mon[38695]: osdmap e46: 7 total, 6 up, 7 in 2024-05-08T20:05:32.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:31 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-08T20:05:32.963 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:32 smithi188 ceph-mon[48761]: purged_snaps scrub starts 2024-05-08T20:05:32.963 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:32 smithi188 ceph-mon[48761]: purged_snaps scrub ok 2024-05-08T20:05:32.963 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:32 smithi188 ceph-mon[48761]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-05-08T20:05:32.963 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:32 smithi188 ceph-mon[48761]: osd.6 [v2:172.21.15.188:6816/118382910,v1:172.21.15.188:6817/118382910] boot 2024-05-08T20:05:32.963 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:32 smithi188 ceph-mon[48761]: osdmap e47: 7 total, 7 up, 7 in 2024-05-08T20:05:32.963 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:32 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-08T20:05:32.963 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:32 smithi188 ceph-mon[48761]: osdmap e48: 7 total, 7 up, 7 in 2024-05-08T20:05:33.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:32 smithi083 ceph-mon[38695]: purged_snaps scrub starts 2024-05-08T20:05:33.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:32 smithi083 ceph-mon[38695]: purged_snaps scrub ok 2024-05-08T20:05:33.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:32 smithi083 ceph-mon[38695]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-05-08T20:05:33.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:32 smithi083 ceph-mon[38695]: osd.6 [v2:172.21.15.188:6816/118382910,v1:172.21.15.188:6817/118382910] boot 2024-05-08T20:05:33.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:32 smithi083 ceph-mon[38695]: osdmap e47: 7 total, 7 up, 7 in 2024-05-08T20:05:33.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:32 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-08T20:05:33.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:32 smithi083 ceph-mon[38695]: osdmap e48: 7 total, 7 up, 7 in 2024-05-08T20:05:34.167 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-08T20:05:34.168 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-08T20:05:34.168 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-05-08T20:05:34.168 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-05-08T20:05:34.168 INFO:teuthology.orchestra.run.smithi188.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0141441 s, 741 MB/s 2024-05-08T20:05:34.168 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-05-08T20:05:35.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:34 smithi188 ceph-mon[48761]: osdmap e49: 7 total, 7 up, 7 in 2024-05-08T20:05:35.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:34 smithi188 ceph-mon[48761]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-08T20:05:35.143 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:34 smithi083 ceph-mon[38695]: osdmap e49: 7 total, 7 up, 7 in 2024-05-08T20:05:35.143 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:34 smithi083 ceph-mon[38695]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-08T20:05:35.640 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_1 2024-05-08T20:05:35.832 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:35 smithi188 ceph-mon[48761]: osdmap e50: 7 total, 7 up, 7 in 2024-05-08T20:05:35.832 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:35.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:35.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:05:35.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:05:35.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:05:35.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:35.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:05:35.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:05:35.833 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:35.870 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi188/config 2024-05-08T20:05:36.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:35 smithi083 ceph-mon[38695]: osdmap e50: 7 total, 7 up, 7 in 2024-05-08T20:05:36.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:36.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:36.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:05:36.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:05:36.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:05:36.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:36.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:05:36.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:05:36.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:37.045 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:36 smithi188 ceph-mon[48761]: Detected new or changed devices on smithi188 2024-05-08T20:05:37.045 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:36 smithi188 ceph-mon[48761]: Adjusting osd_memory_target on smithi188 to 4982M 2024-05-08T20:05:37.045 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:36 smithi188 ceph-mon[48761]: osdmap e51: 7 total, 7 up, 7 in 2024-05-08T20:05:37.045 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:36 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:05:37.045 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:36 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:05:37.046 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:36 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:05:37.046 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:36 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:37.046 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:36 smithi188 ceph-mon[48761]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-05-08T20:05:37.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:36 smithi083 ceph-mon[38695]: Detected new or changed devices on smithi188 2024-05-08T20:05:37.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:36 smithi083 ceph-mon[38695]: Adjusting osd_memory_target on smithi188 to 4982M 2024-05-08T20:05:37.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:36 smithi083 ceph-mon[38695]: osdmap e51: 7 total, 7 up, 7 in 2024-05-08T20:05:37.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:36 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:05:37.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:36 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:05:37.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:36 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:05:37.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:36 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:37.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:36 smithi083 ceph-mon[38695]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-05-08T20:05:38.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:37 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-08T20:05:38.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:37 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-08T20:05:38.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:37 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:05:38.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:37 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-08T20:05:38.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:37 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-08T20:05:38.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:37 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:05:39.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:38 smithi083 ceph-mon[38695]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-05-08T20:05:39.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:38 smithi083 ceph-mon[38695]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:05:39.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:05:39.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:38 smithi188 ceph-mon[48761]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-05-08T20:05:39.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:38 smithi188 ceph-mon[48761]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:05:39.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:05:41.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:40 smithi083 ceph-mon[38695]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-05-08T20:05:41.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:40 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/462157241' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fffd2a6-163d-4a9b-adc2-25329a3803ea"}]: dispatch 2024-05-08T20:05:41.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:40 smithi083 ceph-mon[38695]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fffd2a6-163d-4a9b-adc2-25329a3803ea"}]: dispatch 2024-05-08T20:05:41.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:40 smithi083 ceph-mon[38695]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8fffd2a6-163d-4a9b-adc2-25329a3803ea"}]': finished 2024-05-08T20:05:41.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:40 smithi083 ceph-mon[38695]: osdmap e52: 8 total, 7 up, 8 in 2024-05-08T20:05:41.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:40 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-08T20:05:41.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:40 smithi188 ceph-mon[48761]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-05-08T20:05:41.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:40 smithi188 ceph-mon[48761]: from='client.? 172.21.15.188:0/462157241' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fffd2a6-163d-4a9b-adc2-25329a3803ea"}]: dispatch 2024-05-08T20:05:41.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:40 smithi188 ceph-mon[48761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fffd2a6-163d-4a9b-adc2-25329a3803ea"}]: dispatch 2024-05-08T20:05:41.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:40 smithi188 ceph-mon[48761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8fffd2a6-163d-4a9b-adc2-25329a3803ea"}]': finished 2024-05-08T20:05:41.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:40 smithi188 ceph-mon[48761]: osdmap e52: 8 total, 7 up, 8 in 2024-05-08T20:05:41.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:40 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-08T20:05:42.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:41 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/3604668795' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-08T20:05:42.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:41 smithi188 ceph-mon[48761]: from='client.? 172.21.15.188:0/3604668795' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-08T20:05:43.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:42 smithi083 ceph-mon[38695]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-05-08T20:05:43.311 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:42 smithi188 ceph-mon[48761]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-05-08T20:05:45.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:44 smithi188 ceph-mon[48761]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-08T20:05:45.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:44 smithi083 ceph-mon[38695]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-08T20:05:47.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:46 smithi083 ceph-mon[38695]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-08T20:05:47.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:46 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-08T20:05:47.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:46 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:05:47.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:46 smithi188 ceph-mon[48761]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-08T20:05:47.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:46 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-08T20:05:47.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:46 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:05:48.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:47 smithi083 ceph-mon[38695]: Deploying daemon osd.7 on smithi188 2024-05-08T20:05:48.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:47 smithi188 ceph-mon[48761]: Deploying daemon osd.7 on smithi188 2024-05-08T20:05:49.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:48 smithi188 ceph-mon[48761]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-05-08T20:05:49.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:48 smithi083 ceph-mon[38695]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-05-08T20:05:51.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:50 smithi083 ceph-mon[38695]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-05-08T20:05:51.215 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:50 smithi188 ceph-mon[48761]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-05-08T20:05:53.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:52 smithi083 ceph-mon[38695]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-05-08T20:05:53.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:52 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:53.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:52 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:53.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:52 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:05:53.230 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:52 smithi188 ceph-mon[48761]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-05-08T20:05:53.230 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:52 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:53.230 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:52 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:53.230 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:52 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:05:54.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:53 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:05:54.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:53 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:05:55.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:54 smithi083 ceph-mon[38695]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-05-08T20:05:55.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:54 smithi188 ceph-mon[48761]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-05-08T20:05:56.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:55 smithi188 ceph-mon[48761]: from='osd.7 [v2:172.21.15.188:6824/3349446159,v1:172.21.15.188:6825/3349446159]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-08T20:05:56.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:55 smithi188 ceph-mon[48761]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-08T20:05:56.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:55 smithi083 ceph-mon[38695]: from='osd.7 [v2:172.21.15.188:6824/3349446159,v1:172.21.15.188:6825/3349446159]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-08T20:05:56.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:55 smithi083 ceph-mon[38695]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-08T20:05:57.211 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:56 smithi188 ceph-mon[48761]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-05-08T20:05:57.211 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:56 smithi188 ceph-mon[48761]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-08T20:05:57.211 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:56 smithi188 ceph-mon[48761]: osdmap e53: 8 total, 7 up, 8 in 2024-05-08T20:05:57.211 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:56 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-08T20:05:57.211 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:56 smithi188 ceph-mon[48761]: from='osd.7 [v2:172.21.15.188:6824/3349446159,v1:172.21.15.188:6825/3349446159]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-05-08T20:05:57.211 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:56 smithi188 ceph-mon[48761]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-05-08T20:05:57.349 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 7 on host 'smithi188' 2024-05-08T20:05:57.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:56 smithi083 ceph-mon[38695]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-05-08T20:05:57.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:56 smithi083 ceph-mon[38695]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-08T20:05:57.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:56 smithi083 ceph-mon[38695]: osdmap e53: 8 total, 7 up, 8 in 2024-05-08T20:05:57.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:56 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-08T20:05:57.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:56 smithi083 ceph-mon[38695]: from='osd.7 [v2:172.21.15.188:6824/3349446159,v1:172.21.15.188:6825/3349446159]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-05-08T20:05:57.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:56 smithi083 ceph-mon[38695]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-05-08T20:05:58.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:57 smithi188 ceph-mon[48761]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-05-08T20:05:58.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:57 smithi188 ceph-mon[48761]: osdmap e54: 8 total, 7 up, 8 in 2024-05-08T20:05:58.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-08T20:05:58.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-08T20:05:58.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:58.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:58.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:57 smithi188 ceph-mon[48761]: from='osd.7 ' entity='osd.7' 2024-05-08T20:05:58.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:57 smithi188 ceph-mon[48761]: osd.7 [v2:172.21.15.188:6824/3349446159,v1:172.21.15.188:6825/3349446159] boot 2024-05-08T20:05:58.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:57 smithi188 ceph-mon[48761]: osdmap e55: 8 total, 8 up, 8 in 2024-05-08T20:05:58.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-08T20:05:58.145 DEBUG:teuthology.orchestra.run.smithi188:osd.7> sudo journalctl -f -n 0 -u ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.7.service 2024-05-08T20:05:58.148 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-05-08T20:05:58.148 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph osd stat -f json 2024-05-08T20:05:58.183 INFO:journalctl@ceph.osd.7.smithi188.stdout:-- Logs begin at Wed 2024-05-08 19:48:17 UTC. -- 2024-05-08T20:05:58.380 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:05:58.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:57 smithi083 ceph-mon[38695]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-05-08T20:05:58.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:57 smithi083 ceph-mon[38695]: osdmap e54: 8 total, 7 up, 8 in 2024-05-08T20:05:58.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-08T20:05:58.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-08T20:05:58.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:58.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:58.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:57 smithi083 ceph-mon[38695]: from='osd.7 ' entity='osd.7' 2024-05-08T20:05:58.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:57 smithi083 ceph-mon[38695]: osd.7 [v2:172.21.15.188:6824/3349446159,v1:172.21.15.188:6825/3349446159] boot 2024-05-08T20:05:58.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:57 smithi083 ceph-mon[38695]: osdmap e55: 8 total, 8 up, 8 in 2024-05-08T20:05:58.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-08T20:05:59.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:58 smithi188 ceph-mon[48761]: purged_snaps scrub starts 2024-05-08T20:05:59.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:58 smithi188 ceph-mon[48761]: purged_snaps scrub ok 2024-05-08T20:05:59.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:58 smithi188 ceph-mon[48761]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-05-08T20:05:59.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:58 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:59.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:58 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:59.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:05:58 smithi188 ceph-mon[48761]: osdmap e56: 8 total, 8 up, 8 in 2024-05-08T20:05:59.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:58 smithi083 ceph-mon[38695]: purged_snaps scrub starts 2024-05-08T20:05:59.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:58 smithi083 ceph-mon[38695]: purged_snaps scrub ok 2024-05-08T20:05:59.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:58 smithi083 ceph-mon[38695]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-05-08T20:05:59.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:58 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:59.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:58 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:05:59.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:05:58 smithi083 ceph-mon[38695]: osdmap e56: 8 total, 8 up, 8 in 2024-05-08T20:06:00.403 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:06:01.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:00 smithi188 ceph-mon[48761]: osdmap e57: 8 total, 8 up, 8 in 2024-05-08T20:06:01.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:00 smithi188 ceph-mon[48761]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:01.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:00 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/207332357' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-08T20:06:01.112 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:00 smithi083 ceph-mon[38695]: osdmap e57: 8 total, 8 up, 8 in 2024-05-08T20:06:01.112 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:00 smithi083 ceph-mon[38695]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:01.112 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:00 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/207332357' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-08T20:06:01.131 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":57,"num_osds":8,"num_up_osds":8,"osd_up_since":1715198757,"num_in_osds":8,"osd_in_since":1715198740,"num_remapped_pgs":1} 2024-05-08T20:06:01.132 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-08T20:06:01.358 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:06:01.913 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:01 smithi083 ceph-mon[38695]: osdmap e58: 8 total, 8 up, 8 in 2024-05-08T20:06:02.023 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:01 smithi188 ceph-mon[48761]: osdmap e58: 8 total, 8 up, 8 in 2024-05-08T20:06:03.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:02 smithi188 ceph-mon[48761]: osdmap e59: 8 total, 8 up, 8 in 2024-05-08T20:06:03.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:02 smithi188 ceph-mon[48761]: pgmap v186: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-08T20:06:03.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:02 smithi188 ceph-mon[48761]: Detected new or changed devices on smithi188 2024-05-08T20:06:03.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:06:03.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:06:03.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:06:03.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:06:03.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:06:03.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:06:03.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:02 smithi188 ceph-mon[48761]: Adjusting osd_memory_target on smithi188 to 3736M 2024-05-08T20:06:03.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:06:03.083 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:06:03.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:06:03.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:06:03.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:06:03.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:06:03.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:06:03.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:06:03.137 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:02 smithi083 ceph-mon[38695]: osdmap e59: 8 total, 8 up, 8 in 2024-05-08T20:06:03.137 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:02 smithi083 ceph-mon[38695]: pgmap v186: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-08T20:06:03.137 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:02 smithi083 ceph-mon[38695]: Detected new or changed devices on smithi188 2024-05-08T20:06:03.137 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:06:03.137 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:06:03.137 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:06:03.137 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:06:03.137 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:06:03.137 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:06:03.137 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:02 smithi083 ceph-mon[38695]: Adjusting osd_memory_target on smithi188 to 3736M 2024-05-08T20:06:03.137 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:06:03.137 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:06:03.137 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:06:03.138 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:06:03.138 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:06:03.138 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:06:03.138 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:06:03.138 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:06:03.573 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:06:03.574 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":59,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","created":"2024-05-08T19:59:07.152446+0000","modified":"2024-05-08T20:06:01.775753+0000","last_up_change":"2024-05-08T20:05:57.759893+0000","last_in_change":"2024-05-08T20:05:40.402230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T20:03:44.088148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78051d3d-3644-4876-af5f-8c956a8491bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6803","nonce":345419579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6805","nonce":345419579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6809","nonce":345419579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6807","nonce":345419579}]},"public_addr":"172.21.15.83:6803/345419579","cluster_addr":"172.21.15.83:6805/345419579","heartbeat_back_addr":"172.21.15.83:6809/345419579","heartbeat_front_addr":"172.21.15.83:6807/345419579","state":["exists","up"]},{"osd":1,"uuid":"bc62b8e9-e30e-41f1-aaee-4c4e356be487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6811","nonce":1992633959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6813","nonce":1992633959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6817","nonce":1992633959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6815","nonce":1992633959}]},"public_addr":"172.21.15.83:6811/1992633959","cluster_addr":"172.21.15.83:6813/1992633959","heartbeat_back_addr":"172.21.15.83:6817/1992633959","heartbeat_front_addr":"172.21.15.83:6815/1992633959","state":["exists","up"]},{"osd":2,"uuid":"4ae0453a-718b-445f-ad00-a3696fec1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6819","nonce":561864588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6821","nonce":561864588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6825","nonce":561864588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6823","nonce":561864588}]},"public_addr":"172.21.15.83:6819/561864588","cluster_addr":"172.21.15.83:6821/561864588","heartbeat_back_addr":"172.21.15.83:6825/561864588","heartbeat_front_addr":"172.21.15.83:6823/561864588","state":["exists","up"]},{"osd":3,"uuid":"2085ec85-187a-4d69-a721-793cfed7d044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6827","nonce":866184273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6829","nonce":866184273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6833","nonce":866184273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6831","nonce":866184273}]},"public_addr":"172.21.15.83:6827/866184273","cluster_addr":"172.21.15.83:6829/866184273","heartbeat_back_addr":"172.21.15.83:6833/866184273","heartbeat_front_addr":"172.21.15.83:6831/866184273","state":["exists","up"]},{"osd":4,"uuid":"b92dad8b-5bd0-4cd1-ba8d-2920927f132e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6801","nonce":422247785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6803","nonce":422247785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6807","nonce":422247785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6805","nonce":422247785}]},"public_addr":"172.21.15.188:6801/422247785","cluster_addr":"172.21.15.188:6803/422247785","heartbeat_back_addr":"172.21.15.188:6807/422247785","heartbeat_front_addr":"172.21.15.188:6805/422247785","state":["exists","up"]},{"osd":5,"uuid":"aeeda050-668f-47a2-90d0-32e1aa35bfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6809","nonce":841652631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6811","nonce":841652631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6815","nonce":841652631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6813","nonce":841652631}]},"public_addr":"172.21.15.188:6809/841652631","cluster_addr":"172.21.15.188:6811/841652631","heartbeat_back_addr":"172.21.15.188:6815/841652631","heartbeat_front_addr":"172.21.15.188:6813/841652631","state":["exists","up"]},{"osd":6,"uuid":"d3720c87-ec2b-490a-937b-a29833f0267c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6817","nonce":118382910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6819","nonce":118382910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6823","nonce":118382910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6821","nonce":118382910}]},"public_addr":"172.21.15.188:6817/118382910","cluster_addr":"172.21.15.188:6819/118382910","heartbeat_back_addr":"172.21.15.188:6823/118382910","heartbeat_front_addr":"172.21.15.188:6821/118382910","state":["exists","up"]},{"osd":7,"uuid":"8fffd2a6-163d-4a9b-adc2-25329a3803ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6825","nonce":3349446159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6827","nonce":3349446159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6831","nonce":3349446159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6829","nonce":3349446159}]},"public_addr":"172.21.15.188:6825/3349446159","cluster_addr":"172.21.15.188:6827/3349446159","heartbeat_back_addr":"172.21.15.188:6831/3349446159","heartbeat_front_addr":"172.21.15.188:6829/3349446159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:02:42.828169+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:11.944380+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:41.150631+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:11.648151+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:36.764713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:03.134028+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:29.964086+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:56.788654+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:6801/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3612788753":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3117466271":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/3109058932":"2024-05-09T20:01:23.760420+0000","172.21.15.83:6800/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6800/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/890224504":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/2142540104":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/1765857891":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6801/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6800/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/2981330630":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/2443488799":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6801/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/3512048858":"2024-05-09T19:59:36.209291+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T20:06:03.934 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:03 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/698030987' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:06:04.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:03 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/698030987' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:06:04.307 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T20:03:44.088148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T20:06:04.307 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-08T20:06:04.535 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:06:05.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:04 smithi188 ceph-mon[48761]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-05-08T20:06:05.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:04 smithi083 ceph-mon[38695]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-05-08T20:06:06.543 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-05-08T20:06:07.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:06 smithi188 ceph-mon[48761]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-05-08T20:06:07.082 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:06 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/1243501329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-08T20:06:07.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:06 smithi083 ceph-mon[38695]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-05-08T20:06:07.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:06 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/1243501329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-08T20:06:07.247 INFO:tasks.cephadm:Setting up client nodes... 2024-05-08T20:06:07.248 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-08T20:06:07.475 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:06:08.991 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:08 smithi083 ceph-mon[38695]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-08T20:06:08.991 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:06:09.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:08 smithi188 ceph-mon[48761]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-08T20:06:09.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:06:09.546 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2024-05-08T20:06:09.546 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQAx2ztm66wOIRAAKS7fSXMuszbBGwx0N7MkEg== 2024-05-08T20:06:10.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:09 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/1244365035' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-08T20:06:10.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:09 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/1244365035' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-08T20:06:10.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:09 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/1244365035' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-08T20:06:10.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:09 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/1244365035' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-08T20:06:10.360 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-05-08T20:06:10.360 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-08T20:06:10.360 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-08T20:06:10.399 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-08T20:06:10.639 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi188/config 2024-05-08T20:06:11.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:10 smithi083 ceph-mon[38695]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:11.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:10 smithi188 ceph-mon[48761]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:13.033 INFO:teuthology.orchestra.run.smithi188.stdout:[client.1] 2024-05-08T20:06:13.033 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQA12ztmBIdTAhAAeng6kE5DaZxtzWOXfmyBjg== 2024-05-08T20:06:13.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:12 smithi083 ceph-mon[38695]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:13.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:12 smithi188 ceph-mon[48761]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:13.919 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-05-08T20:06:13.919 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-05-08T20:06:13.919 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-05-08T20:06:13.961 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-08T20:06:13.961 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-08T20:06:13.961 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph mgr dump --format=json 2024-05-08T20:06:14.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:13 smithi083 ceph-mon[38695]: from='client.? 172.21.15.188:0/1323421477' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-08T20:06:14.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:13 smithi083 ceph-mon[38695]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-08T20:06:14.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:13 smithi083 ceph-mon[38695]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-08T20:06:14.201 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:06:14.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:13 smithi188 ceph-mon[48761]: from='client.? 172.21.15.188:0/1323421477' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-08T20:06:14.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:13 smithi188 ceph-mon[48761]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-08T20:06:14.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:13 smithi188 ceph-mon[48761]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-08T20:06:15.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:15 smithi188 ceph-mon[48761]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:15.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:15 smithi083 ceph-mon[38695]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:16.442 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:06:17.097 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:17 smithi083 ceph-mon[38695]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:17.097 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:17 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3823369802' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-08T20:06:17.116 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":19,"flags":0,"active_gid":14211,"active_name":"smithi083.fgcgxm","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":3132406206},{"type":"v1","addr":"172.21.15.83:6801","nonce":3132406206}]},"active_addr":"172.21.15.83:6801/3132406206","active_change":"2024-05-08T20:01:23.760556+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":14234,"name":"smithi188.abfynx","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.83:8443/","prometheus":"http://172.21.15.83:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":3364200811}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":1250063671}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":3710317728}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":339670569}]}]} 2024-05-08T20:06:17.124 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-08T20:06:17.124 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-08T20:06:17.125 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-08T20:06:17.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:17 smithi188 ceph-mon[48761]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:17.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:17 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/3823369802' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-08T20:06:17.349 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:06:19.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:19 smithi083 ceph-mon[38695]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:19.287 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:06:19.287 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":59,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","created":"2024-05-08T19:59:07.152446+0000","modified":"2024-05-08T20:06:01.775753+0000","last_up_change":"2024-05-08T20:05:57.759893+0000","last_in_change":"2024-05-08T20:05:40.402230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T20:03:44.088148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78051d3d-3644-4876-af5f-8c956a8491bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6803","nonce":345419579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6805","nonce":345419579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6809","nonce":345419579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6807","nonce":345419579}]},"public_addr":"172.21.15.83:6803/345419579","cluster_addr":"172.21.15.83:6805/345419579","heartbeat_back_addr":"172.21.15.83:6809/345419579","heartbeat_front_addr":"172.21.15.83:6807/345419579","state":["exists","up"]},{"osd":1,"uuid":"bc62b8e9-e30e-41f1-aaee-4c4e356be487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6811","nonce":1992633959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6813","nonce":1992633959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6817","nonce":1992633959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6815","nonce":1992633959}]},"public_addr":"172.21.15.83:6811/1992633959","cluster_addr":"172.21.15.83:6813/1992633959","heartbeat_back_addr":"172.21.15.83:6817/1992633959","heartbeat_front_addr":"172.21.15.83:6815/1992633959","state":["exists","up"]},{"osd":2,"uuid":"4ae0453a-718b-445f-ad00-a3696fec1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6819","nonce":561864588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6821","nonce":561864588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6825","nonce":561864588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6823","nonce":561864588}]},"public_addr":"172.21.15.83:6819/561864588","cluster_addr":"172.21.15.83:6821/561864588","heartbeat_back_addr":"172.21.15.83:6825/561864588","heartbeat_front_addr":"172.21.15.83:6823/561864588","state":["exists","up"]},{"osd":3,"uuid":"2085ec85-187a-4d69-a721-793cfed7d044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6827","nonce":866184273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6829","nonce":866184273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6833","nonce":866184273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6831","nonce":866184273}]},"public_addr":"172.21.15.83:6827/866184273","cluster_addr":"172.21.15.83:6829/866184273","heartbeat_back_addr":"172.21.15.83:6833/866184273","heartbeat_front_addr":"172.21.15.83:6831/866184273","state":["exists","up"]},{"osd":4,"uuid":"b92dad8b-5bd0-4cd1-ba8d-2920927f132e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6801","nonce":422247785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6803","nonce":422247785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6807","nonce":422247785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6805","nonce":422247785}]},"public_addr":"172.21.15.188:6801/422247785","cluster_addr":"172.21.15.188:6803/422247785","heartbeat_back_addr":"172.21.15.188:6807/422247785","heartbeat_front_addr":"172.21.15.188:6805/422247785","state":["exists","up"]},{"osd":5,"uuid":"aeeda050-668f-47a2-90d0-32e1aa35bfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6809","nonce":841652631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6811","nonce":841652631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6815","nonce":841652631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6813","nonce":841652631}]},"public_addr":"172.21.15.188:6809/841652631","cluster_addr":"172.21.15.188:6811/841652631","heartbeat_back_addr":"172.21.15.188:6815/841652631","heartbeat_front_addr":"172.21.15.188:6813/841652631","state":["exists","up"]},{"osd":6,"uuid":"d3720c87-ec2b-490a-937b-a29833f0267c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6817","nonce":118382910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6819","nonce":118382910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6823","nonce":118382910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6821","nonce":118382910}]},"public_addr":"172.21.15.188:6817/118382910","cluster_addr":"172.21.15.188:6819/118382910","heartbeat_back_addr":"172.21.15.188:6823/118382910","heartbeat_front_addr":"172.21.15.188:6821/118382910","state":["exists","up"]},{"osd":7,"uuid":"8fffd2a6-163d-4a9b-adc2-25329a3803ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6825","nonce":3349446159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6827","nonce":3349446159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6831","nonce":3349446159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6829","nonce":3349446159}]},"public_addr":"172.21.15.188:6825/3349446159","cluster_addr":"172.21.15.188:6827/3349446159","heartbeat_back_addr":"172.21.15.188:6831/3349446159","heartbeat_front_addr":"172.21.15.188:6829/3349446159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:02:42.828169+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:11.944380+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:41.150631+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:11.648151+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:36.764713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:03.134028+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:29.964086+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:56.788654+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:6801/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3612788753":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3117466271":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/3109058932":"2024-05-09T20:01:23.760420+0000","172.21.15.83:6800/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6800/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/890224504":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/2142540104":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/1765857891":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6801/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6800/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/2981330630":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/2443488799":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6801/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/3512048858":"2024-05-09T19:59:36.209291+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T20:06:19.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:19 smithi188 ceph-mon[48761]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:20.000 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-08T20:06:20.000 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-08T20:06:20.227 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:06:20.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:20 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/2800083001' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:06:20.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:20 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2800083001' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:06:21.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:21 smithi188 ceph-mon[48761]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:21.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:21 smithi083 ceph-mon[38695]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:22.166 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:06:22.167 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":59,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","created":"2024-05-08T19:59:07.152446+0000","modified":"2024-05-08T20:06:01.775753+0000","last_up_change":"2024-05-08T20:05:57.759893+0000","last_in_change":"2024-05-08T20:05:40.402230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T20:03:44.088148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78051d3d-3644-4876-af5f-8c956a8491bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6803","nonce":345419579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6805","nonce":345419579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6809","nonce":345419579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6807","nonce":345419579}]},"public_addr":"172.21.15.83:6803/345419579","cluster_addr":"172.21.15.83:6805/345419579","heartbeat_back_addr":"172.21.15.83:6809/345419579","heartbeat_front_addr":"172.21.15.83:6807/345419579","state":["exists","up"]},{"osd":1,"uuid":"bc62b8e9-e30e-41f1-aaee-4c4e356be487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6811","nonce":1992633959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6813","nonce":1992633959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6817","nonce":1992633959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6815","nonce":1992633959}]},"public_addr":"172.21.15.83:6811/1992633959","cluster_addr":"172.21.15.83:6813/1992633959","heartbeat_back_addr":"172.21.15.83:6817/1992633959","heartbeat_front_addr":"172.21.15.83:6815/1992633959","state":["exists","up"]},{"osd":2,"uuid":"4ae0453a-718b-445f-ad00-a3696fec1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6819","nonce":561864588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6821","nonce":561864588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6825","nonce":561864588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6823","nonce":561864588}]},"public_addr":"172.21.15.83:6819/561864588","cluster_addr":"172.21.15.83:6821/561864588","heartbeat_back_addr":"172.21.15.83:6825/561864588","heartbeat_front_addr":"172.21.15.83:6823/561864588","state":["exists","up"]},{"osd":3,"uuid":"2085ec85-187a-4d69-a721-793cfed7d044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6827","nonce":866184273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6829","nonce":866184273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6833","nonce":866184273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6831","nonce":866184273}]},"public_addr":"172.21.15.83:6827/866184273","cluster_addr":"172.21.15.83:6829/866184273","heartbeat_back_addr":"172.21.15.83:6833/866184273","heartbeat_front_addr":"172.21.15.83:6831/866184273","state":["exists","up"]},{"osd":4,"uuid":"b92dad8b-5bd0-4cd1-ba8d-2920927f132e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6801","nonce":422247785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6803","nonce":422247785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6807","nonce":422247785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6805","nonce":422247785}]},"public_addr":"172.21.15.188:6801/422247785","cluster_addr":"172.21.15.188:6803/422247785","heartbeat_back_addr":"172.21.15.188:6807/422247785","heartbeat_front_addr":"172.21.15.188:6805/422247785","state":["exists","up"]},{"osd":5,"uuid":"aeeda050-668f-47a2-90d0-32e1aa35bfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6809","nonce":841652631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6811","nonce":841652631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6815","nonce":841652631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6813","nonce":841652631}]},"public_addr":"172.21.15.188:6809/841652631","cluster_addr":"172.21.15.188:6811/841652631","heartbeat_back_addr":"172.21.15.188:6815/841652631","heartbeat_front_addr":"172.21.15.188:6813/841652631","state":["exists","up"]},{"osd":6,"uuid":"d3720c87-ec2b-490a-937b-a29833f0267c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6817","nonce":118382910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6819","nonce":118382910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6823","nonce":118382910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6821","nonce":118382910}]},"public_addr":"172.21.15.188:6817/118382910","cluster_addr":"172.21.15.188:6819/118382910","heartbeat_back_addr":"172.21.15.188:6823/118382910","heartbeat_front_addr":"172.21.15.188:6821/118382910","state":["exists","up"]},{"osd":7,"uuid":"8fffd2a6-163d-4a9b-adc2-25329a3803ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6825","nonce":3349446159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6827","nonce":3349446159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6831","nonce":3349446159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6829","nonce":3349446159}]},"public_addr":"172.21.15.188:6825/3349446159","cluster_addr":"172.21.15.188:6827/3349446159","heartbeat_back_addr":"172.21.15.188:6831/3349446159","heartbeat_front_addr":"172.21.15.188:6829/3349446159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:02:42.828169+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:11.944380+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:41.150631+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:11.648151+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:36.764713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:03.134028+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:29.964086+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:56.788654+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:6801/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3612788753":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3117466271":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/3109058932":"2024-05-09T20:01:23.760420+0000","172.21.15.83:6800/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6800/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/890224504":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/2142540104":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/1765857891":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6801/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6800/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/2981330630":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/2443488799":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6801/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/3512048858":"2024-05-09T19:59:36.209291+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T20:06:22.816 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-08T20:06:22.817 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-08T20:06:22.817 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-08T20:06:22.818 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-08T20:06:22.818 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-08T20:06:22.819 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-08T20:06:22.819 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-08T20:06:22.819 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-08T20:06:23.070 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:23 smithi083 ceph-mon[38695]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:23.070 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:23 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2410657755' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:06:23.245 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:06:23.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:23 smithi188 ceph-mon[48761]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:23.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:23 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/2410657755' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:06:23.570 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:06:23.573 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:06:23.573 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:06:23.576 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:06:23.576 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:06:23.576 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:06:23.578 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:06:24.169 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:06:24.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:24 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:06:25.182 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:25 smithi083 ceph-mon[38695]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:25.332 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:25 smithi188 ceph-mon[48761]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:27.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:27 smithi188 ceph-mon[48761]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:27.376 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:27 smithi083 ceph-mon[38695]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:29.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:29 smithi083 ceph-mon[38695]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:29.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:29 smithi188 ceph-mon[48761]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:30.142 INFO:teuthology.orchestra.run.smithi083.stdout:111669149726 2024-05-08T20:06:30.143 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-08T20:06:31.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:31 smithi188 ceph-mon[48761]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:31.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:31 smithi083 ceph-mon[38695]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:32.441 INFO:teuthology.orchestra.run.smithi083.stdout:201863462926 2024-05-08T20:06:32.442 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-08T20:06:33.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:33 smithi188 ceph-mon[48761]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:33.432 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:33 smithi083 ceph-mon[38695]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:35.431 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:35 smithi083 ceph-mon[38695]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:35.583 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:35 smithi188 ceph-mon[48761]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:36.560 INFO:teuthology.orchestra.run.smithi083.stdout:38654705713 2024-05-08T20:06:36.561 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-08T20:06:36.994 INFO:teuthology.orchestra.run.smithi083.stdout:236223201290 2024-05-08T20:06:36.994 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-08T20:06:37.324 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:37 smithi083 ceph-mon[38695]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:37.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:37 smithi188 ceph-mon[48761]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:39.125 INFO:teuthology.orchestra.run.smithi083.stdout:133143986203 2024-05-08T20:06:39.126 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-08T20:06:39.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:39 smithi083 ceph-mon[38695]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:39.433 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:06:39.583 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:39 smithi188 ceph-mon[48761]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:39.583 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:06:39.764 INFO:teuthology.orchestra.run.smithi083.stdout:60129542188 2024-05-08T20:06:39.764 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-08T20:06:40.084 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:06:41.113 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:41 smithi083 ceph-mon[38695]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:41.115 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:06:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:06:41.120 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:06:41.124 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:06:41.143 INFO:teuthology.orchestra.run.smithi083.stdout:81604378662 2024-05-08T20:06:41.143 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-08T20:06:41.583 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:41 smithi188 ceph-mon[48761]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:42.288 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:42 smithi083 ceph-mon[38695]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:42.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:42 smithi188 ceph-mon[48761]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:43.084 INFO:teuthology.orchestra.run.smithi083.stdout:111669149729 2024-05-08T20:06:43.107 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:43 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3232652772' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-08T20:06:43.387 INFO:teuthology.orchestra.run.smithi083.stdout:167503724565 2024-05-08T20:06:43.387 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-08T20:06:43.583 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:43 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/3232652772' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-08T20:06:44.082 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149726 got 111669149729 for osd.3 2024-05-08T20:06:44.082 DEBUG:teuthology.parallel:result is None 2024-05-08T20:06:44.336 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:44 smithi083 ceph-mon[38695]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:44.583 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:44 smithi188 ceph-mon[48761]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:46.883 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:06:46.983 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:06:47.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:46 smithi083 ceph-mon[38695]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:47.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:46 smithi188 ceph-mon[48761]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:49.105 INFO:teuthology.orchestra.run.smithi083.stdout:38654705716 2024-05-08T20:06:49.118 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:48 smithi083 ceph-mon[38695]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:49.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:48 smithi188 ceph-mon[48761]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:49.532 INFO:teuthology.orchestra.run.smithi083.stdout:201863462930 2024-05-08T20:06:50.019 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705713 got 38654705716 for osd.0 2024-05-08T20:06:50.019 DEBUG:teuthology.parallel:result is None 2024-05-08T20:06:50.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:49 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2362111969' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-08T20:06:50.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:49 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/843501564' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-08T20:06:50.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:49 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/2362111969' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-08T20:06:50.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:49 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/843501564' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-08T20:06:51.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:50 smithi083 ceph-mon[38695]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:51.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:50 smithi188 ceph-mon[48761]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:51.371 INFO:teuthology.orchestra.run.smithi083.stdout:60129542191 2024-05-08T20:06:51.530 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:06:52.089 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:51 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/558857736' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-08T20:06:52.091 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542188 got 60129542191 for osd.1 2024-05-08T20:06:52.091 DEBUG:teuthology.parallel:result is None 2024-05-08T20:06:52.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:51 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/558857736' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-08T20:06:52.872 INFO:teuthology.orchestra.run.smithi083.stdout:236223201293 2024-05-08T20:06:53.155 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:52 smithi083 ceph-mon[38695]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:53.155 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:52 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3912562201' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-08T20:06:53.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:52 smithi188 ceph-mon[48761]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:53.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:52 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/3912562201' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-08T20:06:54.144 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:53 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:06:54.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:53 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:06:54.972 INFO:teuthology.orchestra.run.smithi083.stdout:133143986206 2024-05-08T20:06:55.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:54 smithi083 ceph-mon[38695]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:55.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:54 smithi188 ceph-mon[48761]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:55.598 INFO:teuthology.orchestra.run.smithi083.stdout:81604378665 2024-05-08T20:06:55.990 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462926 got 201863462930 for osd.6 2024-05-08T20:06:55.991 DEBUG:teuthology.parallel:result is None 2024-05-08T20:06:56.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:55 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/1666094420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-08T20:06:56.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:55 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2548048906' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-08T20:06:56.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:55 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/1666094420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-08T20:06:56.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:55 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/2548048906' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-08T20:06:56.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:56 smithi083 ceph-mon[38695]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:57.167 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:56 smithi188 ceph-mon[48761]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:57.426 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201290 got 236223201293 for osd.7 2024-05-08T20:06:57.426 DEBUG:teuthology.parallel:result is None 2024-05-08T20:06:57.567 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986203 got 133143986206 for osd.4 2024-05-08T20:06:57.567 DEBUG:teuthology.parallel:result is None 2024-05-08T20:06:58.366 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378662 got 81604378665 for osd.2 2024-05-08T20:06:58.366 DEBUG:teuthology.parallel:result is None 2024-05-08T20:06:59.048 INFO:teuthology.orchestra.run.smithi083.stdout:167503724569 2024-05-08T20:06:59.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:58 smithi083 ceph-mon[38695]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:59.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:58 smithi188 ceph-mon[48761]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:06:59.655 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724565 got 167503724569 for osd.5 2024-05-08T20:06:59.655 DEBUG:teuthology.parallel:result is None 2024-05-08T20:06:59.655 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-08T20:06:59.655 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph pg dump --format=json 2024-05-08T20:06:59.884 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:07:00.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:06:59 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/149287359' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-08T20:07:00.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:06:59 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/149287359' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-08T20:07:01.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:00 smithi083 ceph-mon[38695]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:01.333 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:00 smithi188 ceph-mon[48761]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:02.096 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:07:02.098 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-05-08T20:07:02.921 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-05-08T20:07:01.827132+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":230876,"kb_used_data":6500,"kb_used_omap":12,"kb_used_meta":224243,"kb_avail":749730340,"statfs":{"total":767960285184,"available":767723868160,"internally_reserved":0,"allocated":6656000,"data_stored":3738092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":229625400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002413"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":215,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-08T20:06:01.786143+0000","last_change":"2024-05-08T20:06:01.783575+0000","last_active":"2024-05-08T20:06:01.786143+0000","last_peered":"2024-05-08T20:06:01.786143+0000","last_clean":"2024-05-08T20:06:01.786143+0000","last_became_active":"2024-05-08T20:06:01.783203+0000","last_became_peered":"2024-05-08T20:06:01.783203+0000","last_unstale":"2024-05-08T20:06:01.786143+0000","last_undegraded":"2024-05-08T20:06:01.786143+0000","last_fullsized":"2024-05-08T20:06:01.786143+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T20:03:44.502886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T20:03:44.502886+0000","last_clean_scrub_stamp":"2024-05-08T20:03:44.502886+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:48:18.459171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201295,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28196,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716956,"statfs":{"total":95995035648,"available":95966162944,"internally_reserved":0,"allocated":1204224,"data_stored":836317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]}]},{"osd":6,"up_from":47,"seq":201863462932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28200,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716952,"statfs":{"total":95995035648,"available":95966158848,"internally_reserved":0,"allocated":1200128,"data_stored":836044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.495},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.47599999999999998}]},{"osd":1,"last update":"Wed May 8 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.39700000000000002}]},{"osd":2,"last update":"Wed May 8 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.51200000000000001}]},{"osd":3,"last update":"Wed May 8 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.442}]},{"osd":4,"last update":"Wed May 8 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.45200000000000001}]},{"osd":5,"last update":"Wed May 8 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.60699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542193,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27616,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717536,"statfs":{"total":95995035648,"available":95966756864,"internally_reserved":0,"allocated":610304,"data_stored":245949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 20:06:21 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.33300000000000002,"15min":0.307},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.32500000000000001,"15min":0.30299999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.40100000000000002}]},{"osd":2,"last update":"Wed May 8 20:06:44 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.35899999999999999,"15min":0.34599999999999997},"min":{"1min":0.246,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.51300000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.35999999999999999,"15min":0.34200000000000003},"min":{"1min":0.24099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.499,"5min":0.624,"15min":0.624},"last":0.36699999999999999}]},{"osd":3,"last update":"Wed May 8 20:06:21 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.44600000000000001,"15min":0.44500000000000001},"min":{"1min":0.26000000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.434},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.441,"15min":0.439},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.42099999999999999}]},{"osd":4,"last update":"Wed May 8 20:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.86399999999999999}]},{"osd":5,"last update":"Wed May 8 20:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.84299999999999997}]},{"osd":6,"last update":"Wed May 8 20:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.80100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32292,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712860,"statfs":{"total":95995035648,"available":95961968640,"internally_reserved":0,"allocated":1204224,"data_stored":836317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed May 8 20:06:23 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.32700000000000001,"15min":0.29799999999999999},"min":{"1min":0.24099999999999999,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.375},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.33100000000000002,"15min":0.29699999999999999},"min":{"1min":0.28000000000000003,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.54400000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.38900000000000001}]},{"osd":2,"last update":"Wed May 8 20:06:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.36299999999999999,"15min":0.35499999999999998},"min":{"1min":0.23999999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.46000000000000002,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.38600000000000001,"15min":0.371},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.55200000000000005,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.45900000000000002}]},{"osd":3,"last update":"Wed May 8 20:06:23 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.45900000000000002,"15min":0.45800000000000002},"min":{"1min":0.33600000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.44400000000000001,"15min":0.44900000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.42999999999999999}]},{"osd":4,"last update":"Wed May 8 20:06:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.57299999999999995,"15min":0.57899999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.68899999999999995,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61599999999999999,"15min":0.623},"min":{"1min":0.46200000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.72299999999999998,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.53600000000000003}]},{"osd":5,"last update":"Wed May 8 20:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.57999999999999996}]},{"osd":6,"last update":"Wed May 8 20:06:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.59299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]},{"osd":2,"up_from":19,"seq":81604378667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31712,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713440,"statfs":{"total":95995035648,"available":95962562560,"internally_reserved":0,"allocated":610304,"data_stored":245949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 20:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.36699999999999999,"15min":0.36499999999999999},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.501,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.5},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.38700000000000001,"15min":0.38800000000000001},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.56499999999999995,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.52500000000000002}]},{"osd":1,"last update":"Wed May 8 20:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.38400000000000001,"15min":0.38600000000000001},"min":{"1min":0.252,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.621,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.375,"15min":0.371},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.54500000000000004}]},{"osd":3,"last update":"Wed May 8 20:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45200000000000001,"15min":0.45400000000000001},"min":{"1min":0.32100000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.44500000000000001,"15min":0.44800000000000001},"min":{"1min":0.30299999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.57899999999999996}]},{"osd":4,"last update":"Wed May 8 20:06:40 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.44900000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82299999999999995,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.60299999999999998,"15min":0.59699999999999998},"min":{"1min":0.46800000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.85199999999999998,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.75}]},{"osd":5,"last update":"Wed May 8 20:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.68400000000000005}]},{"osd":6,"last update":"Wed May 8 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.71099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]}]},{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27616,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717536,"statfs":{"total":95995035648,"available":95966756864,"internally_reserved":0,"allocated":610304,"data_stored":245949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 20:06:18 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.40600000000000003,"15min":0.40999999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.51000000000000001,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.312},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42999999999999999,"15min":0.43099999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.54400000000000004,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.35199999999999998}]},{"osd":1,"last update":"Wed May 8 20:06:18 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.42599999999999999,"15min":0.42399999999999999},"min":{"1min":0.312,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.63800000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.42899999999999999,"15min":0.43099999999999999},"min":{"1min":0.29799999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.32000000000000001}]},{"osd":2,"last update":"Wed May 8 20:06:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.63200000000000001,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.441,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.313,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.60999999999999999,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.27900000000000003}]},{"osd":4,"last update":"Wed May 8 20:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.74199999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.45000000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77800000000000002,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.58799999999999997}]},{"osd":5,"last update":"Wed May 8 20:06:08 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.62},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.77700000000000002}]},{"osd":6,"last update":"Wed May 8 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.745}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]}]},{"osd":4,"up_from":31,"seq":133143986207,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27624,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717528,"statfs":{"total":95995035648,"available":95966748672,"internally_reserved":0,"allocated":610304,"data_stored":245949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 20:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.61399999999999999,"15min":0.626},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.67400000000000004,"5min":1.97,"15min":1.97},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.66000000000000003,"15min":0.67400000000000004},"min":{"1min":0.41199999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.72499999999999998,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":0.72099999999999997}]},{"osd":1,"last update":"Wed May 8 20:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.61399999999999999,"15min":0.63},"min":{"1min":0.33900000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.64700000000000002,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.60199999999999998,"15min":0.61299999999999999},"min":{"1min":0.433,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.71699999999999997,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":0.60599999999999998}]},{"osd":2,"last update":"Wed May 8 20:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.60999999999999999,"15min":0.621},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.73999999999999999,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.65700000000000003,"15min":0.67200000000000004},"min":{"1min":0.42499999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.73099999999999998,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.67300000000000004}]},{"osd":3,"last update":"Wed May 8 20:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.63600000000000001,"15min":0.65000000000000002},"min":{"1min":0.36599999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70699999999999996,"5min":1.998,"15min":1.998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.629,"15min":0.64300000000000002},"min":{"1min":0.40000000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.69399999999999995,"5min":1.8049999999999999,"15min":1.8049999999999999},"last":0.66000000000000003}]},{"osd":5,"last update":"Wed May 8 20:06:09 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.75}]},{"osd":6,"last update":"Wed May 8 20:06:34 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.68500000000000005}]},{"osd":7,"last update":"Wed May 8 20:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.69399999999999995}]}]},{"osd":5,"up_from":39,"seq":167503724570,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27620,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717532,"statfs":{"total":95995035648,"available":95966752768,"internally_reserved":0,"allocated":606208,"data_stored":245618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 20:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.71999999999999997}]},{"osd":1,"last update":"Wed May 8 20:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.74299999999999999}]},{"osd":2,"last update":"Wed May 8 20:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.70799999999999996}]},{"osd":3,"last update":"Wed May 8 20:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.66400000000000003}]},{"osd":4,"last update":"Wed May 8 20:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.625},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.57099999999999995}]},{"osd":6,"last update":"Wed May 8 20:06:34 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.73499999999999999}]},{"osd":7,"last update":"Wed May 8 20:06:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.77000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T20:07:02.923 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph pg dump --format=json 2024-05-08T20:07:03.158 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:07:03.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:02 smithi083 ceph-mon[38695]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:03.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:02 smithi083 ceph-mon[38695]: from='client.14468 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-08T20:07:03.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:02 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:07:03.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:02 smithi188 ceph-mon[48761]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:03.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:02 smithi188 ceph-mon[48761]: from='client.14468 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-08T20:07:03.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:02 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:07:04.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:03 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:07:04.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:03 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:07:04.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:03 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:04.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:03 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:07:04.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:03 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:07:04.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:03 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:05.183 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:04 smithi083 ceph-mon[38695]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:05.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:04 smithi188 ceph-mon[48761]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:05.339 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:07:05.341 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-05-08T20:07:06.084 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2024-05-08T20:07:03.827412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":230876,"kb_used_data":6500,"kb_used_omap":12,"kb_used_meta":224243,"kb_avail":749730340,"statfs":{"total":767960285184,"available":767723868160,"internally_reserved":0,"allocated":6656000,"data_stored":3738092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":229625400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002302"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":215,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-08T20:06:01.786143+0000","last_change":"2024-05-08T20:06:01.783575+0000","last_active":"2024-05-08T20:06:01.786143+0000","last_peered":"2024-05-08T20:06:01.786143+0000","last_clean":"2024-05-08T20:06:01.786143+0000","last_became_active":"2024-05-08T20:06:01.783203+0000","last_became_peered":"2024-05-08T20:06:01.783203+0000","last_unstale":"2024-05-08T20:06:01.786143+0000","last_undegraded":"2024-05-08T20:06:01.786143+0000","last_fullsized":"2024-05-08T20:06:01.786143+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T20:03:44.502886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T20:03:44.502886+0000","last_clean_scrub_stamp":"2024-05-08T20:03:44.502886+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:48:18.459171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201296,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28196,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716956,"statfs":{"total":95995035648,"available":95966162944,"internally_reserved":0,"allocated":1204224,"data_stored":836317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 20:06:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.44},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.3079999999999998,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":0.628}]},{"osd":1,"last update":"Wed May 8 20:06:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.38},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.55700000000000005}]},{"osd":2,"last update":"Wed May 8 20:06:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":3.1349999999999998,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":7.516,"5min":7.516,"15min":7.516},"last":0.59999999999999998}]},{"osd":3,"last update":"Wed May 8 20:06:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.66100000000000003}]},{"osd":4,"last update":"Wed May 8 20:06:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.57999999999999996}]},{"osd":5,"last update":"Wed May 8 20:06:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.70899999999999996}]},{"osd":6,"last update":"Wed May 8 20:06:58 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.69299999999999995}]}]},{"osd":6,"up_from":47,"seq":201863462933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28200,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716952,"statfs":{"total":95995035648,"available":95966158848,"internally_reserved":0,"allocated":1200128,"data_stored":836044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.49099999999999999}]},{"osd":1,"last update":"Wed May 8 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.502},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.41899999999999998}]},{"osd":2,"last update":"Wed May 8 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.375},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.55800000000000005}]},{"osd":3,"last update":"Wed May 8 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.46200000000000002}]},{"osd":4,"last update":"Wed May 8 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.53100000000000003}]},{"osd":5,"last update":"Wed May 8 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.47899999999999998}]},{"osd":7,"last update":"Wed May 8 20:06:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.55100000000000005}]}]},{"osd":1,"up_from":14,"seq":60129542193,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27616,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717536,"statfs":{"total":95995035648,"available":95966756864,"internally_reserved":0,"allocated":610304,"data_stored":245949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 20:06:21 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.33300000000000002,"15min":0.307},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.32500000000000001,"15min":0.30299999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.40100000000000002}]},{"osd":2,"last update":"Wed May 8 20:06:44 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.35899999999999999,"15min":0.34599999999999997},"min":{"1min":0.246,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.51300000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.35999999999999999,"15min":0.34200000000000003},"min":{"1min":0.24099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.499,"5min":0.624,"15min":0.624},"last":0.36699999999999999}]},{"osd":3,"last update":"Wed May 8 20:06:21 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.44600000000000001,"15min":0.44500000000000001},"min":{"1min":0.26000000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.434},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.441,"15min":0.439},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.42099999999999999}]},{"osd":4,"last update":"Wed May 8 20:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.86399999999999999}]},{"osd":5,"last update":"Wed May 8 20:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.84299999999999997}]},{"osd":6,"last update":"Wed May 8 20:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.80100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32292,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712860,"statfs":{"total":95995035648,"available":95961968640,"internally_reserved":0,"allocated":1204224,"data_stored":836317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed May 8 20:06:23 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.32700000000000001,"15min":0.29799999999999999},"min":{"1min":0.24099999999999999,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.375},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.33100000000000002,"15min":0.29699999999999999},"min":{"1min":0.28000000000000003,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.54400000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.38900000000000001}]},{"osd":2,"last update":"Wed May 8 20:06:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.36299999999999999,"15min":0.35499999999999998},"min":{"1min":0.23999999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.46000000000000002,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.38600000000000001,"15min":0.371},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.55200000000000005,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.45900000000000002}]},{"osd":3,"last update":"Wed May 8 20:06:23 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.45900000000000002,"15min":0.45800000000000002},"min":{"1min":0.33600000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.44400000000000001,"15min":0.44900000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.42999999999999999}]},{"osd":4,"last update":"Wed May 8 20:06:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.57299999999999995,"15min":0.57899999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.68899999999999995,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61599999999999999,"15min":0.623},"min":{"1min":0.46200000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.72299999999999998,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.53600000000000003}]},{"osd":5,"last update":"Wed May 8 20:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.57999999999999996}]},{"osd":6,"last update":"Wed May 8 20:06:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.59299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]},{"osd":2,"up_from":19,"seq":81604378667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31712,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713440,"statfs":{"total":95995035648,"available":95962562560,"internally_reserved":0,"allocated":610304,"data_stored":245949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 20:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.36699999999999999,"15min":0.36499999999999999},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.501,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.5},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.38700000000000001,"15min":0.38800000000000001},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.56499999999999995,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.52500000000000002}]},{"osd":1,"last update":"Wed May 8 20:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.38400000000000001,"15min":0.38600000000000001},"min":{"1min":0.252,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.621,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.375,"15min":0.371},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.54500000000000004}]},{"osd":3,"last update":"Wed May 8 20:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45200000000000001,"15min":0.45400000000000001},"min":{"1min":0.32100000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.44500000000000001,"15min":0.44800000000000001},"min":{"1min":0.30299999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.57899999999999996}]},{"osd":4,"last update":"Wed May 8 20:06:40 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.44900000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82299999999999995,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.60299999999999998,"15min":0.59699999999999998},"min":{"1min":0.46800000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.85199999999999998,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.75}]},{"osd":5,"last update":"Wed May 8 20:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.68400000000000005}]},{"osd":6,"last update":"Wed May 8 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.71099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]}]},{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27616,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717536,"statfs":{"total":95995035648,"available":95966756864,"internally_reserved":0,"allocated":610304,"data_stored":245949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 20:06:18 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.40600000000000003,"15min":0.40999999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.51000000000000001,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.312},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42999999999999999,"15min":0.43099999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.54400000000000004,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.35199999999999998}]},{"osd":1,"last update":"Wed May 8 20:06:18 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.42599999999999999,"15min":0.42399999999999999},"min":{"1min":0.312,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.63800000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.42899999999999999,"15min":0.43099999999999999},"min":{"1min":0.29799999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.32000000000000001}]},{"osd":2,"last update":"Wed May 8 20:06:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.63200000000000001,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.441,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.313,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.60999999999999999,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.27900000000000003}]},{"osd":4,"last update":"Wed May 8 20:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.74199999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.45000000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77800000000000002,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.58799999999999997}]},{"osd":5,"last update":"Wed May 8 20:06:08 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.62},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.77700000000000002}]},{"osd":6,"last update":"Wed May 8 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.745}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]}]},{"osd":4,"up_from":31,"seq":133143986208,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27624,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717528,"statfs":{"total":95995035648,"available":95966748672,"internally_reserved":0,"allocated":610304,"data_stored":245949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 20:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.61399999999999999,"15min":0.626},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.67400000000000004,"5min":1.97,"15min":1.97},"last":0.436},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.66000000000000003,"15min":0.67400000000000004},"min":{"1min":0.41199999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.72499999999999998,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":0.55200000000000005}]},{"osd":1,"last update":"Wed May 8 20:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.61399999999999999,"15min":0.63},"min":{"1min":0.33900000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.64700000000000002,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.60199999999999998,"15min":0.61299999999999999},"min":{"1min":0.433,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.71699999999999997,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":0.54300000000000004}]},{"osd":2,"last update":"Wed May 8 20:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.60999999999999999,"15min":0.621},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.73999999999999999,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.65700000000000003,"15min":0.67200000000000004},"min":{"1min":0.42499999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.73099999999999998,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.58899999999999997}]},{"osd":3,"last update":"Wed May 8 20:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.63600000000000001,"15min":0.65000000000000002},"min":{"1min":0.36599999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70699999999999996,"5min":1.998,"15min":1.998},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.629,"15min":0.64300000000000002},"min":{"1min":0.40000000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.69399999999999995,"5min":1.8049999999999999,"15min":1.8049999999999999},"last":0.52600000000000002}]},{"osd":5,"last update":"Wed May 8 20:06:09 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.37},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.61099999999999999}]},{"osd":6,"last update":"Wed May 8 20:06:34 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.499}]},{"osd":7,"last update":"Wed May 8 20:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.56799999999999995}]}]},{"osd":5,"up_from":39,"seq":167503724570,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27620,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717532,"statfs":{"total":95995035648,"available":95966752768,"internally_reserved":0,"allocated":606208,"data_stored":245618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed May 8 20:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.71999999999999997}]},{"osd":1,"last update":"Wed May 8 20:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.74299999999999999}]},{"osd":2,"last update":"Wed May 8 20:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.70799999999999996}]},{"osd":3,"last update":"Wed May 8 20:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.66400000000000003}]},{"osd":4,"last update":"Wed May 8 20:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.625},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.57099999999999995}]},{"osd":6,"last update":"Wed May 8 20:06:34 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.73499999999999999}]},{"osd":7,"last update":"Wed May 8 20:06:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.77000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T20:07:06.086 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-08T20:07:06.086 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-08T20:07:06.086 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-08T20:07:06.086 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph health --format=json 2024-05-08T20:07:06.312 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:07:07.166 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:06 smithi083 ceph-mon[38695]: from='client.14472 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-08T20:07:07.166 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:06 smithi083 ceph-mon[38695]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:07.265 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:06 smithi188 ceph-mon[48761]: from='client.14472 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-08T20:07:07.265 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:06 smithi188 ceph-mon[48761]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:08.464 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:07:08.464 INFO:teuthology.orchestra.run.smithi083.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-08T20:07:09.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:08 smithi188 ceph-mon[48761]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:09.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:08 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/1521011883' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-08T20:07:09.121 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:08 smithi083 ceph-mon[38695]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:09.121 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:08 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/1521011883' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-08T20:07:09.142 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-08T20:07:09.142 INFO:tasks.cephadm:Setup complete, yielding 2024-05-08T20:07:09.143 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-08T20:07:09.153 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi083.front.sepia.ceph.com 2024-05-08T20:07:09.153 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- bash -c 'ceph orch status' 2024-05-08T20:07:09.382 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:07:09.933 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:07:10.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:07:11.050 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:10 smithi083 ceph-mon[38695]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:11.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:10 smithi188 ceph-mon[48761]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:11.566 INFO:teuthology.orchestra.run.smithi083.stdout:Backend: cephadm 2024-05-08T20:07:11.566 INFO:teuthology.orchestra.run.smithi083.stdout:Available: Yes 2024-05-08T20:07:11.566 INFO:teuthology.orchestra.run.smithi083.stdout:Paused: No 2024-05-08T20:07:12.281 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- bash -c 'ceph orch ps' 2024-05-08T20:07:12.507 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:07:13.001 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:12 smithi083 ceph-mon[38695]: from='client.14480 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:07:13.002 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:12 smithi083 ceph-mon[38695]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:13.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:12 smithi188 ceph-mon[48761]: from='client.14480 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:07:13.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:12 smithi188 ceph-mon[48761]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:14.799 INFO:teuthology.orchestra.run.smithi083.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-08T20:07:14.799 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager.smithi083 smithi083 *:9093,9094 running (4m) 2m ago 6m 26.7M - 0.25.0 c8568f914cd2 23e6da64785c 2024-05-08T20:07:14.800 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-exporter.smithi083 smithi083 running (6m) 2m ago 6m 7071k - 18.2.2-1322-gd21e4f96 af23193ff691 4292591c8fb3 2024-05-08T20:07:14.800 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-exporter.smithi188 smithi188 running (5m) 76s ago 5m 7096k - 18.2.2-1322-gd21e4f96 af23193ff691 cf99700fb705 2024-05-08T20:07:14.800 INFO:teuthology.orchestra.run.smithi083.stdout:crash.smithi083 smithi083 running (6m) 2m ago 6m 6656k - 18.2.2-1322-gd21e4f96 af23193ff691 25e825846282 2024-05-08T20:07:14.800 INFO:teuthology.orchestra.run.smithi083.stdout:crash.smithi188 smithi188 running (5m) 76s ago 5m 6651k - 18.2.2-1322-gd21e4f96 af23193ff691 4d47f2875338 2024-05-08T20:07:14.800 INFO:teuthology.orchestra.run.smithi083.stdout:grafana.smithi083 smithi083 *:3000 running (4m) 2m ago 6m 80.7M - 9.4.7 954c08fa6188 c661dfb9dd01 2024-05-08T20:07:14.800 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.smithi083.fgcgxm smithi083 *:9283,8765,8443 running (8m) 2m ago 8m 493M - 18.2.2-1322-gd21e4f96 af23193ff691 5fadb826661f 2024-05-08T20:07:14.800 INFO:teuthology.orchestra.run.smithi083.stdout:mgr.smithi188.abfynx smithi188 *:8443,9283,8765 running (5m) 76s ago 5m 450M - 18.2.2-1322-gd21e4f96 af23193ff691 1e522457e762 2024-05-08T20:07:14.800 INFO:teuthology.orchestra.run.smithi083.stdout:mon.smithi083 smithi083 running (8m) 2m ago 8m 46.1M 2048M 18.2.2-1322-gd21e4f96 af23193ff691 7b4a52f78f65 2024-05-08T20:07:14.800 INFO:teuthology.orchestra.run.smithi083.stdout:mon.smithi188 smithi188 running (5m) 76s ago 5m 45.1M 2048M 18.2.2-1322-gd21e4f96 af23193ff691 3d6556ae54b1 2024-05-08T20:07:14.800 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.smithi083 smithi083 *:9100 running (6m) 2m ago 6m 17.5M - 1.5.0 0da6a335fe13 d71d53477cac 2024-05-08T20:07:14.800 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter.smithi188 smithi188 *:9100 running (5m) 76s ago 5m 15.4M - 1.5.0 0da6a335fe13 4303d3cf8829 2024-05-08T20:07:14.800 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 smithi083 running (4m) 2m ago 4m 45.9M 2968M 18.2.2-1322-gd21e4f96 af23193ff691 cc4938bbe6cd 2024-05-08T20:07:14.800 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 smithi083 running (4m) 2m ago 4m 37.1M 2968M 18.2.2-1322-gd21e4f96 af23193ff691 75b73f2baca4 2024-05-08T20:07:14.801 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 smithi083 running (3m) 2m ago 3m 42.9M 2968M 18.2.2-1322-gd21e4f96 af23193ff691 a83963e12417 2024-05-08T20:07:14.801 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 smithi083 running (3m) 2m ago 3m 11.4M 2968M 18.2.2-1322-gd21e4f96 af23193ff691 301be2979048 2024-05-08T20:07:14.801 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 smithi188 running (2m) 76s ago 2m 37.2M 3736M 18.2.2-1322-gd21e4f96 af23193ff691 5ad785a92925 2024-05-08T20:07:14.801 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 smithi188 running (2m) 76s ago 2m 37.6M 3736M 18.2.2-1322-gd21e4f96 af23193ff691 382c3c7a6386 2024-05-08T20:07:14.801 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 smithi188 running (109s) 76s ago 109s 33.7M 3736M 18.2.2-1322-gd21e4f96 af23193ff691 4bfb9bc99876 2024-05-08T20:07:14.801 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 smithi188 running (82s) 76s ago 82s 12.7M 3736M 18.2.2-1322-gd21e4f96 af23193ff691 342228de8915 2024-05-08T20:07:14.801 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus.smithi083 smithi083 *:9095 running (4m) 2m ago 5m 45.0M - 2.43.0 a07b618ecd1d 74191e57d068 2024-05-08T20:07:15.129 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:14 smithi083 ceph-mon[38695]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:15.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:14 smithi188 ceph-mon[48761]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:15.453 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- bash -c 'ceph orch ls' 2024-05-08T20:07:15.679 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:07:16.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:15 smithi083 ceph-mon[38695]: from='client.14484 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:07:16.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:15 smithi188 ceph-mon[48761]: from='client.14484 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:07:17.046 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:16 smithi083 ceph-mon[38695]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:17.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:16 smithi188 ceph-mon[48761]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:17.805 INFO:teuthology.orchestra.run.smithi083.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-05-08T20:07:17.805 INFO:teuthology.orchestra.run.smithi083.stdout:alertmanager ?:9093,9094 1/1 3m ago 7m count:1 2024-05-08T20:07:17.806 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-exporter 2/2 3m ago 7m * 2024-05-08T20:07:17.806 INFO:teuthology.orchestra.run.smithi083.stdout:crash 2/2 3m ago 7m * 2024-05-08T20:07:17.806 INFO:teuthology.orchestra.run.smithi083.stdout:grafana ?:3000 1/1 3m ago 7m count:1 2024-05-08T20:07:17.806 INFO:teuthology.orchestra.run.smithi083.stdout:mgr 2/2 3m ago 7m count:2 2024-05-08T20:07:17.806 INFO:teuthology.orchestra.run.smithi083.stdout:mon 2/2 3m ago 6m smithi083:172.21.15.83=smithi083;smithi188:172.21.15.188=smithi188;count:2 2024-05-08T20:07:17.806 INFO:teuthology.orchestra.run.smithi083.stdout:node-exporter ?:9100 2/2 3m ago 7m * 2024-05-08T20:07:17.806 INFO:teuthology.orchestra.run.smithi083.stdout:osd 8 3m ago - 2024-05-08T20:07:17.806 INFO:teuthology.orchestra.run.smithi083.stdout:prometheus ?:9095 1/1 3m ago 7m count:1 2024-05-08T20:07:18.627 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- bash -c 'ceph orch host ls' 2024-05-08T20:07:18.856 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:07:19.143 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:18 smithi083 ceph-mon[38695]: from='client.14488 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:07:19.143 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:18 smithi083 ceph-mon[38695]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:19.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:18 smithi188 ceph-mon[48761]: from='client.14488 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:07:19.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:18 smithi188 ceph-mon[48761]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:21.101 INFO:teuthology.orchestra.run.smithi083.stdout:HOST ADDR LABELS STATUS 2024-05-08T20:07:21.102 INFO:teuthology.orchestra.run.smithi083.stdout:smithi083 172.21.15.83 2024-05-08T20:07:21.102 INFO:teuthology.orchestra.run.smithi083.stdout:smithi188 172.21.15.188 2024-05-08T20:07:21.102 INFO:teuthology.orchestra.run.smithi083.stdout:2 hosts in cluster 2024-05-08T20:07:21.143 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:20 smithi083 ceph-mon[38695]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:21.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:20 smithi188 ceph-mon[48761]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:21.797 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- bash -c 'ceph orch device ls' 2024-05-08T20:07:22.026 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:07:23.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:22 smithi083 ceph-mon[38695]: from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:07:23.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:22 smithi083 ceph-mon[38695]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:23.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:22 smithi188 ceph-mon[48761]: from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:07:23.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:22 smithi188 ceph-mon[48761]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:23.934 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:07:24.141 INFO:teuthology.orchestra.run.smithi083.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-05-08T20:07:24.141 INFO:teuthology.orchestra.run.smithi083.stdout:smithi083 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62040148400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-05-08T20:07:24.141 INFO:teuthology.orchestra.run.smithi083.stdout:smithi188 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500MP400BGN 372G No 82s ago Has a FileSystem, LVM detected 2024-05-08T20:07:24.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:07:25.051 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:24 smithi083 ceph-mon[38695]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:25.073 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-08T20:07:25.082 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi083.front.sepia.ceph.com 2024-05-08T20:07:25.083 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- bash -c 'ceph fs volume create foo' 2024-05-08T20:07:25.316 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:07:25.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:24 smithi188 ceph-mon[48761]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:26.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:25 smithi083 ceph-mon[38695]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:07:26.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:25 smithi188 ceph-mon[48761]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:07:26.934 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:26 smithi083 ceph-mon[38695]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:27.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:26 smithi188 ceph-mon[48761]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:28.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:27 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-05-08T20:07:28.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:27 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-05-08T20:07:29.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:28 smithi083 ceph-mon[38695]: from='client.14500 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:07:29.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:28 smithi083 ceph-mon[38695]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:29.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:28 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-05-08T20:07:29.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:28 smithi083 ceph-mon[38695]: osdmap e60: 8 total, 8 up, 8 in 2024-05-08T20:07:29.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:28 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-05-08T20:07:29.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:28 smithi083 ceph-526038a6-0d75-11ef-bc97-c7b262605968-mon-smithi083[38672]: 2024-05-08T20:07:28.925+0000 7f02998ab700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-08T20:07:29.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:28 smithi188 ceph-mon[48761]: from='client.14500 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:07:29.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:28 smithi188 ceph-mon[48761]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:29.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:28 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-05-08T20:07:29.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:28 smithi188 ceph-mon[48761]: osdmap e60: 8 total, 8 up, 8 in 2024-05-08T20:07:29.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:28 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-05-08T20:07:29.943 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- bash -c 'ceph fs volume create bar' 2024-05-08T20:07:30.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:29 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-05-08T20:07:30.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:29 smithi083 ceph-mon[38695]: osdmap e61: 8 total, 8 up, 8 in 2024-05-08T20:07:30.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:29 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-05-08T20:07:30.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:29 smithi083 ceph-mon[38695]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-08T20:07:30.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:29 smithi083 ceph-mon[38695]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-08T20:07:30.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:29 smithi083 ceph-mon[38695]: osdmap e62: 8 total, 8 up, 8 in 2024-05-08T20:07:30.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:29 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-05-08T20:07:30.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:29 smithi083 ceph-mon[38695]: fsmap foo:0 2024-05-08T20:07:30.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:29 smithi083 ceph-mon[38695]: Saving service mds.foo spec with placement count:2 2024-05-08T20:07:30.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:29 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:30.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:29 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:07:30.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:29 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:07:30.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:29 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:07:30.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:29 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:30.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:29 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi083.dfkbdq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-08T20:07:30.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:29 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi083.dfkbdq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-08T20:07:30.186 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:29 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:07:30.186 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:29 smithi083 ceph-mon[38695]: Deploying daemon mds.foo.smithi083.dfkbdq on smithi083 2024-05-08T20:07:30.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:29 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-05-08T20:07:30.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:29 smithi188 ceph-mon[48761]: osdmap e61: 8 total, 8 up, 8 in 2024-05-08T20:07:30.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:29 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-05-08T20:07:30.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:29 smithi188 ceph-mon[48761]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-08T20:07:30.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:29 smithi188 ceph-mon[48761]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-08T20:07:30.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:29 smithi188 ceph-mon[48761]: osdmap e62: 8 total, 8 up, 8 in 2024-05-08T20:07:30.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:29 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-05-08T20:07:30.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:29 smithi188 ceph-mon[48761]: fsmap foo:0 2024-05-08T20:07:30.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:29 smithi188 ceph-mon[48761]: Saving service mds.foo spec with placement count:2 2024-05-08T20:07:30.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:29 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:30.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:29 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:07:30.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:29 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:07:30.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:29 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:07:30.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:29 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:30.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:29 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi083.dfkbdq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-08T20:07:30.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:29 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi083.dfkbdq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-08T20:07:30.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:29 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:07:30.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:29 smithi188 ceph-mon[48761]: Deploying daemon mds.foo.smithi083.dfkbdq on smithi083 2024-05-08T20:07:31.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:30 smithi083 ceph-mon[38695]: pgmap v233: 65 pgs: 16 creating+peering, 35 unknown, 14 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:31.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:30 smithi083 ceph-mon[38695]: osdmap e63: 8 total, 8 up, 8 in 2024-05-08T20:07:31.235 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:07:31.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:30 smithi188 ceph-mon[48761]: pgmap v233: 65 pgs: 16 creating+peering, 35 unknown, 14 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:31.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:30 smithi188 ceph-mon[48761]: osdmap e63: 8 total, 8 up, 8 in 2024-05-08T20:07:32.250 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:31 smithi083 ceph-mon[38695]: osdmap e64: 8 total, 8 up, 8 in 2024-05-08T20:07:32.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:31 smithi188 ceph-mon[48761]: osdmap e64: 8 total, 8 up, 8 in 2024-05-08T20:07:33.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:32 smithi083 ceph-mon[38695]: pgmap v236: 65 pgs: 22 creating+peering, 17 unknown, 26 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:33.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:32 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:33.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:32 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:33.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:32 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:33.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:32 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi188.tcwiuw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-08T20:07:33.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:32 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi188.tcwiuw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-08T20:07:33.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:32 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:07:33.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:32 smithi083 ceph-mon[38695]: daemon mds.foo.smithi083.dfkbdq assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-05-08T20:07:33.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:32 smithi083 ceph-mon[38695]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-08T20:07:33.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:32 smithi083 ceph-mon[38695]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-08T20:07:33.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:32 smithi083 ceph-mon[38695]: Cluster is now healthy 2024-05-08T20:07:33.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:32 smithi083 ceph-mon[38695]: mds.? [v2:172.21.15.83:6834/486534342,v1:172.21.15.83:6835/486534342] up:boot 2024-05-08T20:07:33.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:32 smithi083 ceph-mon[38695]: fsmap foo:1 {0=foo.smithi083.dfkbdq=up:creating} 2024-05-08T20:07:33.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:32 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mds metadata", "who": "foo.smithi083.dfkbdq"}]: dispatch 2024-05-08T20:07:33.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:32 smithi083 ceph-mon[38695]: daemon mds.foo.smithi083.dfkbdq is now active in filesystem foo as rank 0 2024-05-08T20:07:33.265 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:32 smithi188 ceph-mon[48761]: pgmap v236: 65 pgs: 22 creating+peering, 17 unknown, 26 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:07:33.265 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:32 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:33.265 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:32 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:33.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:32 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:33.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:32 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi188.tcwiuw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-08T20:07:33.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:32 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi188.tcwiuw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-08T20:07:33.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:32 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:07:33.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:32 smithi188 ceph-mon[48761]: daemon mds.foo.smithi083.dfkbdq assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-05-08T20:07:33.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:32 smithi188 ceph-mon[48761]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-08T20:07:33.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:32 smithi188 ceph-mon[48761]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-08T20:07:33.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:32 smithi188 ceph-mon[48761]: Cluster is now healthy 2024-05-08T20:07:33.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:32 smithi188 ceph-mon[48761]: mds.? [v2:172.21.15.83:6834/486534342,v1:172.21.15.83:6835/486534342] up:boot 2024-05-08T20:07:33.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:32 smithi188 ceph-mon[48761]: fsmap foo:1 {0=foo.smithi083.dfkbdq=up:creating} 2024-05-08T20:07:33.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:32 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mds metadata", "who": "foo.smithi083.dfkbdq"}]: dispatch 2024-05-08T20:07:33.266 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:32 smithi188 ceph-mon[48761]: daemon mds.foo.smithi083.dfkbdq is now active in filesystem foo as rank 0 2024-05-08T20:07:34.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:33 smithi188 ceph-mon[48761]: Deploying daemon mds.foo.smithi188.tcwiuw on smithi188 2024-05-08T20:07:34.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:33 smithi188 ceph-mon[48761]: mds.? [v2:172.21.15.83:6834/486534342,v1:172.21.15.83:6835/486534342] up:active 2024-05-08T20:07:34.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:33 smithi188 ceph-mon[48761]: fsmap foo:1 {0=foo.smithi083.dfkbdq=up:active} 2024-05-08T20:07:34.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:33 smithi083 ceph-mon[38695]: Deploying daemon mds.foo.smithi188.tcwiuw on smithi188 2024-05-08T20:07:34.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:33 smithi083 ceph-mon[38695]: mds.? [v2:172.21.15.83:6834/486534342,v1:172.21.15.83:6835/486534342] up:active 2024-05-08T20:07:34.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:33 smithi083 ceph-mon[38695]: fsmap foo:1 {0=foo.smithi083.dfkbdq=up:active} 2024-05-08T20:07:35.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:34 smithi188 ceph-mon[48761]: pgmap v237: 65 pgs: 15 creating+peering, 50 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 208 B/s wr, 1 op/s 2024-05-08T20:07:35.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:34 smithi188 ceph-mon[48761]: from='client.14508 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "bar", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:07:35.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:34 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd pool create", "pool": "cephfs.bar.meta"}]: dispatch 2024-05-08T20:07:35.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:34 smithi083 ceph-mon[38695]: pgmap v237: 65 pgs: 15 creating+peering, 50 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 208 B/s wr, 1 op/s 2024-05-08T20:07:35.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:34 smithi083 ceph-mon[38695]: from='client.14508 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "bar", "target": ["mon-mgr", ""]}]: dispatch 2024-05-08T20:07:35.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:34 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd pool create", "pool": "cephfs.bar.meta"}]: dispatch 2024-05-08T20:07:36.293 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "osd pool create", "pool": "cephfs.bar.meta"}]': finished 2024-05-08T20:07:36.293 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:35 smithi083 ceph-mon[38695]: osdmap e65: 8 total, 8 up, 8 in 2024-05-08T20:07:36.293 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.bar.data"}]: dispatch 2024-05-08T20:07:36.293 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:36.293 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:36.293 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:36.293 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:36.293 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:36.294 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:07:36.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "osd pool create", "pool": "cephfs.bar.meta"}]': finished 2024-05-08T20:07:36.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:35 smithi188 ceph-mon[48761]: osdmap e65: 8 total, 8 up, 8 in 2024-05-08T20:07:36.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.bar.data"}]: dispatch 2024-05-08T20:07:36.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:36.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:36.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:36.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:36.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:36.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:07:36.826 INFO:teuthology.run_tasks:Running task fs.ready... 2024-05-08T20:07:36.837 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T20:07:37.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:36 smithi188 ceph-mon[48761]: pgmap v239: 97 pgs: 2 creating+peering, 30 unknown, 65 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s wr, 5 op/s 2024-05-08T20:07:37.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:36 smithi188 ceph-mon[48761]: mds.? [v2:172.21.15.188:6832/1664076600,v1:172.21.15.188:6833/1664076600] up:boot 2024-05-08T20:07:37.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:36 smithi188 ceph-mon[48761]: fsmap foo:1 {0=foo.smithi083.dfkbdq=up:active} 1 up:standby 2024-05-08T20:07:37.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:36 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mds metadata", "who": "foo.smithi188.tcwiuw"}]: dispatch 2024-05-08T20:07:37.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:36 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.bar.data"}]': finished 2024-05-08T20:07:37.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:36 smithi188 ceph-mon[48761]: osdmap e66: 8 total, 8 up, 8 in 2024-05-08T20:07:37.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:36 smithi188 ceph-mon[48761]: fsmap foo:1 {0=foo.smithi083.dfkbdq=up:active} 1 up:standby 2024-05-08T20:07:37.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:36 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "fs new", "fs_name": "bar", "metadata": "cephfs.bar.meta", "data": "cephfs.bar.data"}]: dispatch 2024-05-08T20:07:37.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:36 smithi188 ceph-mon[48761]: daemon mds.foo.smithi188.tcwiuw assigned to filesystem bar as rank 0 2024-05-08T20:07:37.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:36 smithi188 ceph-mon[48761]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-05-08T20:07:37.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:36 smithi188 ceph-mon[48761]: osdmap e67: 8 total, 8 up, 8 in 2024-05-08T20:07:37.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:36 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "fs new", "fs_name": "bar", "metadata": "cephfs.bar.meta", "data": "cephfs.bar.data"}]': finished 2024-05-08T20:07:37.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:36 smithi188 ceph-mon[48761]: fsmap foo:1 bar:1 {bar:0=foo.smithi188.tcwiuw=up:creating,foo:0=foo.smithi083.dfkbdq=up:active} 2024-05-08T20:07:37.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:36 smithi188 ceph-mon[48761]: Saving service mds.bar spec with placement count:2 2024-05-08T20:07:37.086 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:36 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:37.086 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:36 smithi188 ceph-mon[48761]: daemon mds.foo.smithi188.tcwiuw is now active in filesystem bar as rank 0 2024-05-08T20:07:37.086 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:36 smithi188 ceph-mon[48761]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-08T20:07:37.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:36 smithi083 ceph-mon[38695]: pgmap v239: 97 pgs: 2 creating+peering, 30 unknown, 65 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s wr, 5 op/s 2024-05-08T20:07:37.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:36 smithi083 ceph-mon[38695]: mds.? [v2:172.21.15.188:6832/1664076600,v1:172.21.15.188:6833/1664076600] up:boot 2024-05-08T20:07:37.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:36 smithi083 ceph-mon[38695]: fsmap foo:1 {0=foo.smithi083.dfkbdq=up:active} 1 up:standby 2024-05-08T20:07:37.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:36 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mds metadata", "who": "foo.smithi188.tcwiuw"}]: dispatch 2024-05-08T20:07:37.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:36 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.bar.data"}]': finished 2024-05-08T20:07:37.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:36 smithi083 ceph-mon[38695]: osdmap e66: 8 total, 8 up, 8 in 2024-05-08T20:07:37.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:36 smithi083 ceph-mon[38695]: fsmap foo:1 {0=foo.smithi083.dfkbdq=up:active} 1 up:standby 2024-05-08T20:07:37.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:36 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "fs new", "fs_name": "bar", "metadata": "cephfs.bar.meta", "data": "cephfs.bar.data"}]: dispatch 2024-05-08T20:07:37.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:36 smithi083 ceph-mon[38695]: daemon mds.foo.smithi188.tcwiuw assigned to filesystem bar as rank 0 2024-05-08T20:07:37.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:36 smithi083 ceph-mon[38695]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-05-08T20:07:37.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:36 smithi083 ceph-mon[38695]: osdmap e67: 8 total, 8 up, 8 in 2024-05-08T20:07:37.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:36 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "fs new", "fs_name": "bar", "metadata": "cephfs.bar.meta", "data": "cephfs.bar.data"}]': finished 2024-05-08T20:07:37.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:36 smithi083 ceph-mon[38695]: fsmap foo:1 bar:1 {bar:0=foo.smithi188.tcwiuw=up:creating,foo:0=foo.smithi083.dfkbdq=up:active} 2024-05-08T20:07:37.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:36 smithi083 ceph-mon[38695]: Saving service mds.bar spec with placement count:2 2024-05-08T20:07:37.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:36 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:37.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:36 smithi083 ceph-mon[38695]: daemon mds.foo.smithi188.tcwiuw is now active in filesystem bar as rank 0 2024-05-08T20:07:37.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:36 smithi083 ceph-mon[38695]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-08T20:07:37.200 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:07:37.200 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":68,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","created":"2024-05-08T19:59:07.152446+0000","modified":"2024-05-08T20:07:36.977927+0000","last_up_change":"2024-05-08T20:05:57.759893+0000","last_in_change":"2024-05-08T20:05:40.402230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T20:03:44.088148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.foo.meta","create_time":"2024-05-08T20:07:27.766436+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.data","create_time":"2024-05-08T20:07:28.151095+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.bar.meta","create_time":"2024-05-08T20:07:34.236025+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bar"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs.bar.data","create_time":"2024-05-08T20:07:35.187905+0000","flags":163841,"flags_names":"hashpspool,creating,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bar"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78051d3d-3644-4876-af5f-8c956a8491bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6803","nonce":345419579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6805","nonce":345419579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6809","nonce":345419579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6807","nonce":345419579}]},"public_addr":"172.21.15.83:6803/345419579","cluster_addr":"172.21.15.83:6805/345419579","heartbeat_back_addr":"172.21.15.83:6809/345419579","heartbeat_front_addr":"172.21.15.83:6807/345419579","state":["exists","up"]},{"osd":1,"uuid":"bc62b8e9-e30e-41f1-aaee-4c4e356be487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6811","nonce":1992633959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6813","nonce":1992633959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6817","nonce":1992633959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6815","nonce":1992633959}]},"public_addr":"172.21.15.83:6811/1992633959","cluster_addr":"172.21.15.83:6813/1992633959","heartbeat_back_addr":"172.21.15.83:6817/1992633959","heartbeat_front_addr":"172.21.15.83:6815/1992633959","state":["exists","up"]},{"osd":2,"uuid":"4ae0453a-718b-445f-ad00-a3696fec1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6819","nonce":561864588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6821","nonce":561864588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6825","nonce":561864588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6823","nonce":561864588}]},"public_addr":"172.21.15.83:6819/561864588","cluster_addr":"172.21.15.83:6821/561864588","heartbeat_back_addr":"172.21.15.83:6825/561864588","heartbeat_front_addr":"172.21.15.83:6823/561864588","state":["exists","up"]},{"osd":3,"uuid":"2085ec85-187a-4d69-a721-793cfed7d044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6827","nonce":866184273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6829","nonce":866184273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6833","nonce":866184273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6831","nonce":866184273}]},"public_addr":"172.21.15.83:6827/866184273","cluster_addr":"172.21.15.83:6829/866184273","heartbeat_back_addr":"172.21.15.83:6833/866184273","heartbeat_front_addr":"172.21.15.83:6831/866184273","state":["exists","up"]},{"osd":4,"uuid":"b92dad8b-5bd0-4cd1-ba8d-2920927f132e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6801","nonce":422247785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6803","nonce":422247785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6807","nonce":422247785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6805","nonce":422247785}]},"public_addr":"172.21.15.188:6801/422247785","cluster_addr":"172.21.15.188:6803/422247785","heartbeat_back_addr":"172.21.15.188:6807/422247785","heartbeat_front_addr":"172.21.15.188:6805/422247785","state":["exists","up"]},{"osd":5,"uuid":"aeeda050-668f-47a2-90d0-32e1aa35bfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6809","nonce":841652631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6811","nonce":841652631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6815","nonce":841652631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6813","nonce":841652631}]},"public_addr":"172.21.15.188:6809/841652631","cluster_addr":"172.21.15.188:6811/841652631","heartbeat_back_addr":"172.21.15.188:6815/841652631","heartbeat_front_addr":"172.21.15.188:6813/841652631","state":["exists","up"]},{"osd":6,"uuid":"d3720c87-ec2b-490a-937b-a29833f0267c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6817","nonce":118382910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6819","nonce":118382910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6823","nonce":118382910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6821","nonce":118382910}]},"public_addr":"172.21.15.188:6817/118382910","cluster_addr":"172.21.15.188:6819/118382910","heartbeat_back_addr":"172.21.15.188:6823/118382910","heartbeat_front_addr":"172.21.15.188:6821/118382910","state":["exists","up"]},{"osd":7,"uuid":"8fffd2a6-163d-4a9b-adc2-25329a3803ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6825","nonce":3349446159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6827","nonce":3349446159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6831","nonce":3349446159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6829","nonce":3349446159}]},"public_addr":"172.21.15.188:6825/3349446159","cluster_addr":"172.21.15.188:6827/3349446159","heartbeat_back_addr":"172.21.15.188:6831/3349446159","heartbeat_front_addr":"172.21.15.188:6829/3349446159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:02:42.828169+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:11.944380+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:41.150631+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:11.648151+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:36.764713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:03.134028+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:29.964086+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:56.788654+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:6801/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3612788753":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3117466271":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/3109058932":"2024-05-09T20:01:23.760420+0000","172.21.15.83:6800/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6800/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/890224504":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/2142540104":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/1765857891":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6801/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6800/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/2981330630":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/2443488799":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6801/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/3512048858":"2024-05-09T19:59:36.209291+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T20:07:37.216 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T20:03:44.088148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.foo.meta', 'create_time': '2024-05-08T20:07:27.766436+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.foo.data', 'create_time': '2024-05-08T20:07:28.151095+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs.bar.meta', 'create_time': '2024-05-08T20:07:34.236025+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'bar'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs.bar.data', 'create_time': '2024-05-08T20:07:35.187905+0000', 'flags': 163841, 'flags_names': 'hashpspool,creating,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'bar'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T20:07:37.217 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T20:07:37.563 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-05-08T20:07:37.578 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.meta pg_num 2024-05-08T20:07:37.918 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:07:37.932 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.data pg_num 2024-05-08T20:07:38.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:37 smithi083 ceph-mon[38695]: osdmap e68: 8 total, 8 up, 8 in 2024-05-08T20:07:38.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:37 smithi083 ceph-mon[38695]: mds.? [v2:172.21.15.188:6832/1664076600,v1:172.21.15.188:6833/1664076600] up:active 2024-05-08T20:07:38.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:37 smithi083 ceph-mon[38695]: mds.? [v2:172.21.15.83:6834/486534342,v1:172.21.15.83:6835/486534342] up:active 2024-05-08T20:07:38.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:37 smithi083 ceph-mon[38695]: fsmap foo:1 bar:1 {bar:0=foo.smithi188.tcwiuw=up:active,foo:0=foo.smithi083.dfkbdq=up:active} 2024-05-08T20:07:38.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:37 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/424800801' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:07:38.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:37 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/461052340' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-08T20:07:38.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:37 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:38.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:37 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:38.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:37 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3683173094' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:07:38.280 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:07:38.294 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.meta pg_num 2024-05-08T20:07:38.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:37 smithi188 ceph-mon[48761]: osdmap e68: 8 total, 8 up, 8 in 2024-05-08T20:07:38.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:37 smithi188 ceph-mon[48761]: mds.? [v2:172.21.15.188:6832/1664076600,v1:172.21.15.188:6833/1664076600] up:active 2024-05-08T20:07:38.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:37 smithi188 ceph-mon[48761]: mds.? [v2:172.21.15.83:6834/486534342,v1:172.21.15.83:6835/486534342] up:active 2024-05-08T20:07:38.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:37 smithi188 ceph-mon[48761]: fsmap foo:1 bar:1 {bar:0=foo.smithi188.tcwiuw=up:active,foo:0=foo.smithi083.dfkbdq=up:active} 2024-05-08T20:07:38.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:37 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/424800801' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:07:38.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:37 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/461052340' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-08T20:07:38.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:37 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:38.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:37 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:38.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:37 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/3683173094' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:07:38.643 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:07:38.659 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.data pg_num 2024-05-08T20:07:39.007 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:07:39.023 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T20:07:39.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:38 smithi188 ceph-mon[48761]: pgmap v243: 129 pgs: 5 creating+peering, 56 unknown, 68 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s wr, 9 op/s 2024-05-08T20:07:39.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:38 smithi188 ceph-mon[48761]: osdmap e69: 8 total, 8 up, 8 in 2024-05-08T20:07:39.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:38 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/3268834387' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-05-08T20:07:39.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:38 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/3515570496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:07:39.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:38 smithi188 ceph-mon[48761]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-08T20:07:39.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:39.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:07:39.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:39.404 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:07:39.405 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T20:07:28.926384+0000","modified":"2024-05-08T20:07:36.993402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14504},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14504":{"gid":14504,"name":"foo.smithi083.dfkbdq","rank":0,"incarnation":3,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/486534342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":486534342},{"type":"v1","addr":"172.21.15.83:6835","nonce":486534342}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"foo","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1},{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T20:07:35.967161+0000","modified":"2024-05-08T20:07:36.993402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24273":{"gid":24273,"name":"foo.smithi188.tcwiuw","rank":0,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.188:6833/1664076600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1664076600},{"type":"v1","addr":"172.21.15.188:6833","nonce":1664076600}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"bar","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-05-08T20:07:39.407 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 8 2024-05-08T20:07:39.418 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:38 smithi083 ceph-mon[38695]: pgmap v243: 129 pgs: 5 creating+peering, 56 unknown, 68 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s wr, 9 op/s 2024-05-08T20:07:39.418 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:38 smithi083 ceph-mon[38695]: osdmap e69: 8 total, 8 up, 8 in 2024-05-08T20:07:39.419 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:38 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3268834387' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-05-08T20:07:39.419 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:38 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3515570496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:07:39.419 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:38 smithi083 ceph-mon[38695]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-08T20:07:39.419 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:39.419 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:07:39.419 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:39.420 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T20:07:39.755 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:07:39.755 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":69,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","created":"2024-05-08T19:59:07.152446+0000","modified":"2024-05-08T20:07:37.979695+0000","last_up_change":"2024-05-08T20:05:57.759893+0000","last_in_change":"2024-05-08T20:05:40.402230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T20:03:44.088148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.foo.meta","create_time":"2024-05-08T20:07:27.766436+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.data","create_time":"2024-05-08T20:07:28.151095+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.bar.meta","create_time":"2024-05-08T20:07:34.236025+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bar"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs.bar.data","create_time":"2024-05-08T20:07:35.187905+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bar"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78051d3d-3644-4876-af5f-8c956a8491bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6803","nonce":345419579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6805","nonce":345419579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6809","nonce":345419579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6807","nonce":345419579}]},"public_addr":"172.21.15.83:6803/345419579","cluster_addr":"172.21.15.83:6805/345419579","heartbeat_back_addr":"172.21.15.83:6809/345419579","heartbeat_front_addr":"172.21.15.83:6807/345419579","state":["exists","up"]},{"osd":1,"uuid":"bc62b8e9-e30e-41f1-aaee-4c4e356be487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6811","nonce":1992633959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6813","nonce":1992633959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6817","nonce":1992633959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6815","nonce":1992633959}]},"public_addr":"172.21.15.83:6811/1992633959","cluster_addr":"172.21.15.83:6813/1992633959","heartbeat_back_addr":"172.21.15.83:6817/1992633959","heartbeat_front_addr":"172.21.15.83:6815/1992633959","state":["exists","up"]},{"osd":2,"uuid":"4ae0453a-718b-445f-ad00-a3696fec1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6819","nonce":561864588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6821","nonce":561864588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6825","nonce":561864588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6823","nonce":561864588}]},"public_addr":"172.21.15.83:6819/561864588","cluster_addr":"172.21.15.83:6821/561864588","heartbeat_back_addr":"172.21.15.83:6825/561864588","heartbeat_front_addr":"172.21.15.83:6823/561864588","state":["exists","up"]},{"osd":3,"uuid":"2085ec85-187a-4d69-a721-793cfed7d044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6827","nonce":866184273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6829","nonce":866184273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6833","nonce":866184273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6831","nonce":866184273}]},"public_addr":"172.21.15.83:6827/866184273","cluster_addr":"172.21.15.83:6829/866184273","heartbeat_back_addr":"172.21.15.83:6833/866184273","heartbeat_front_addr":"172.21.15.83:6831/866184273","state":["exists","up"]},{"osd":4,"uuid":"b92dad8b-5bd0-4cd1-ba8d-2920927f132e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6801","nonce":422247785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6803","nonce":422247785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6807","nonce":422247785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6805","nonce":422247785}]},"public_addr":"172.21.15.188:6801/422247785","cluster_addr":"172.21.15.188:6803/422247785","heartbeat_back_addr":"172.21.15.188:6807/422247785","heartbeat_front_addr":"172.21.15.188:6805/422247785","state":["exists","up"]},{"osd":5,"uuid":"aeeda050-668f-47a2-90d0-32e1aa35bfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6809","nonce":841652631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6811","nonce":841652631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6815","nonce":841652631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6813","nonce":841652631}]},"public_addr":"172.21.15.188:6809/841652631","cluster_addr":"172.21.15.188:6811/841652631","heartbeat_back_addr":"172.21.15.188:6815/841652631","heartbeat_front_addr":"172.21.15.188:6813/841652631","state":["exists","up"]},{"osd":6,"uuid":"d3720c87-ec2b-490a-937b-a29833f0267c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6817","nonce":118382910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6819","nonce":118382910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6823","nonce":118382910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6821","nonce":118382910}]},"public_addr":"172.21.15.188:6817/118382910","cluster_addr":"172.21.15.188:6819/118382910","heartbeat_back_addr":"172.21.15.188:6823/118382910","heartbeat_front_addr":"172.21.15.188:6821/118382910","state":["exists","up"]},{"osd":7,"uuid":"8fffd2a6-163d-4a9b-adc2-25329a3803ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6825","nonce":3349446159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6827","nonce":3349446159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6831","nonce":3349446159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6829","nonce":3349446159}]},"public_addr":"172.21.15.188:6825/3349446159","cluster_addr":"172.21.15.188:6827/3349446159","heartbeat_back_addr":"172.21.15.188:6831/3349446159","heartbeat_front_addr":"172.21.15.188:6829/3349446159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:02:42.828169+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:11.944380+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:41.150631+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:11.648151+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:36.764713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:03.134028+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:29.964086+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:56.788654+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:6801/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3612788753":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3117466271":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/3109058932":"2024-05-09T20:01:23.760420+0000","172.21.15.83:6800/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6800/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/890224504":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/2142540104":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/1765857891":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6801/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6800/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/2981330630":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/2443488799":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6801/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/3512048858":"2024-05-09T19:59:36.209291+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T20:07:39.769 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T20:03:44.088148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.foo.meta', 'create_time': '2024-05-08T20:07:27.766436+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.foo.data', 'create_time': '2024-05-08T20:07:28.151095+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs.bar.meta', 'create_time': '2024-05-08T20:07:34.236025+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'bar'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs.bar.data', 'create_time': '2024-05-08T20:07:35.187905+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'bar'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T20:07:39.769 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T20:07:40.109 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-05-08T20:07:40.123 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:39 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/575569153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-05-08T20:07:40.123 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:39 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2796363853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-08T20:07:40.123 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:39 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/845990287' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:07:40.124 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.meta pg_num 2024-05-08T20:07:40.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:39 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/575569153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-05-08T20:07:40.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:39 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/2796363853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-08T20:07:40.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:39 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/845990287' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:07:40.465 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:07:40.480 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.data pg_num 2024-05-08T20:07:40.827 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:07:40.844 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.meta pg_num 2024-05-08T20:07:41.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:40 smithi083 ceph-mon[38695]: pgmap v245: 129 pgs: 5 creating+peering, 13 unknown, 111 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s wr, 12 op/s 2024-05-08T20:07:41.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:40 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/387235238' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-08T20:07:41.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:40 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/1664329158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:07:41.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:40 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:41.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:40 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2385319972' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-05-08T20:07:41.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:40 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:41.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:40 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:07:41.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:40 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:07:41.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:40 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:41.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:40 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.bar.smithi083.vqxrol", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-08T20:07:41.186 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:40 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.bar.smithi083.vqxrol", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-08T20:07:41.186 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:40 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:07:41.199 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:07:41.213 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.data pg_num 2024-05-08T20:07:41.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:40 smithi188 ceph-mon[48761]: pgmap v245: 129 pgs: 5 creating+peering, 13 unknown, 111 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s wr, 12 op/s 2024-05-08T20:07:41.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:40 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/387235238' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-08T20:07:41.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:40 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/1664329158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:07:41.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:40 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:41.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:40 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/2385319972' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-05-08T20:07:41.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:40 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:41.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:40 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:07:41.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:40 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:07:41.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:40 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:41.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:40 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.bar.smithi083.vqxrol", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-08T20:07:41.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:40 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.bar.smithi083.vqxrol", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-08T20:07:41.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:40 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:07:41.558 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:07:41.577 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T20:07:41.951 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:07:41.951 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T20:07:28.926384+0000","modified":"2024-05-08T20:07:36.993402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14504},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14504":{"gid":14504,"name":"foo.smithi083.dfkbdq","rank":0,"incarnation":3,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/486534342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":486534342},{"type":"v1","addr":"172.21.15.83:6835","nonce":486534342}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"foo","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1},{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T20:07:35.967161+0000","modified":"2024-05-08T20:07:36.993402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24273":{"gid":24273,"name":"foo.smithi188.tcwiuw","rank":0,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.188:6833/1664076600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1664076600},{"type":"v1","addr":"172.21.15.188:6833","nonce":1664076600}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"bar","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-05-08T20:07:41.954 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 8 2024-05-08T20:07:41.965 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T20:07:42.300 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:07:42.300 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":69,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","created":"2024-05-08T19:59:07.152446+0000","modified":"2024-05-08T20:07:37.979695+0000","last_up_change":"2024-05-08T20:05:57.759893+0000","last_in_change":"2024-05-08T20:05:40.402230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T20:03:44.088148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.foo.meta","create_time":"2024-05-08T20:07:27.766436+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.data","create_time":"2024-05-08T20:07:28.151095+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.bar.meta","create_time":"2024-05-08T20:07:34.236025+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bar"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs.bar.data","create_time":"2024-05-08T20:07:35.187905+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bar"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78051d3d-3644-4876-af5f-8c956a8491bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6803","nonce":345419579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6805","nonce":345419579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6809","nonce":345419579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6807","nonce":345419579}]},"public_addr":"172.21.15.83:6803/345419579","cluster_addr":"172.21.15.83:6805/345419579","heartbeat_back_addr":"172.21.15.83:6809/345419579","heartbeat_front_addr":"172.21.15.83:6807/345419579","state":["exists","up"]},{"osd":1,"uuid":"bc62b8e9-e30e-41f1-aaee-4c4e356be487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6811","nonce":1992633959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6813","nonce":1992633959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6817","nonce":1992633959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6815","nonce":1992633959}]},"public_addr":"172.21.15.83:6811/1992633959","cluster_addr":"172.21.15.83:6813/1992633959","heartbeat_back_addr":"172.21.15.83:6817/1992633959","heartbeat_front_addr":"172.21.15.83:6815/1992633959","state":["exists","up"]},{"osd":2,"uuid":"4ae0453a-718b-445f-ad00-a3696fec1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6819","nonce":561864588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6821","nonce":561864588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6825","nonce":561864588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6823","nonce":561864588}]},"public_addr":"172.21.15.83:6819/561864588","cluster_addr":"172.21.15.83:6821/561864588","heartbeat_back_addr":"172.21.15.83:6825/561864588","heartbeat_front_addr":"172.21.15.83:6823/561864588","state":["exists","up"]},{"osd":3,"uuid":"2085ec85-187a-4d69-a721-793cfed7d044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6827","nonce":866184273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6829","nonce":866184273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6833","nonce":866184273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6831","nonce":866184273}]},"public_addr":"172.21.15.83:6827/866184273","cluster_addr":"172.21.15.83:6829/866184273","heartbeat_back_addr":"172.21.15.83:6833/866184273","heartbeat_front_addr":"172.21.15.83:6831/866184273","state":["exists","up"]},{"osd":4,"uuid":"b92dad8b-5bd0-4cd1-ba8d-2920927f132e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6801","nonce":422247785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6803","nonce":422247785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6807","nonce":422247785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6805","nonce":422247785}]},"public_addr":"172.21.15.188:6801/422247785","cluster_addr":"172.21.15.188:6803/422247785","heartbeat_back_addr":"172.21.15.188:6807/422247785","heartbeat_front_addr":"172.21.15.188:6805/422247785","state":["exists","up"]},{"osd":5,"uuid":"aeeda050-668f-47a2-90d0-32e1aa35bfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6809","nonce":841652631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6811","nonce":841652631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6815","nonce":841652631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6813","nonce":841652631}]},"public_addr":"172.21.15.188:6809/841652631","cluster_addr":"172.21.15.188:6811/841652631","heartbeat_back_addr":"172.21.15.188:6815/841652631","heartbeat_front_addr":"172.21.15.188:6813/841652631","state":["exists","up"]},{"osd":6,"uuid":"d3720c87-ec2b-490a-937b-a29833f0267c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6817","nonce":118382910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6819","nonce":118382910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6823","nonce":118382910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6821","nonce":118382910}]},"public_addr":"172.21.15.188:6817/118382910","cluster_addr":"172.21.15.188:6819/118382910","heartbeat_back_addr":"172.21.15.188:6823/118382910","heartbeat_front_addr":"172.21.15.188:6821/118382910","state":["exists","up"]},{"osd":7,"uuid":"8fffd2a6-163d-4a9b-adc2-25329a3803ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6825","nonce":3349446159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6827","nonce":3349446159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6831","nonce":3349446159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6829","nonce":3349446159}]},"public_addr":"172.21.15.188:6825/3349446159","cluster_addr":"172.21.15.188:6827/3349446159","heartbeat_back_addr":"172.21.15.188:6831/3349446159","heartbeat_front_addr":"172.21.15.188:6829/3349446159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:02:42.828169+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:11.944380+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:41.150631+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:11.648151+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:36.764713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:03.134028+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:29.964086+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:56.788654+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:6801/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3612788753":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3117466271":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/3109058932":"2024-05-09T20:01:23.760420+0000","172.21.15.83:6800/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6800/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/890224504":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/2142540104":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/1765857891":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6801/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6800/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/2981330630":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/2443488799":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6801/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/3512048858":"2024-05-09T19:59:36.209291+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T20:07:42.313 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:41 smithi083 ceph-mon[38695]: Deploying daemon mds.bar.smithi083.vqxrol on smithi083 2024-05-08T20:07:42.313 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:41 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/1272564396' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:07:42.313 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:41 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/1878635949' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-05-08T20:07:42.313 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:41 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/90474399' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-08T20:07:42.315 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-08T20:07:28.926384+0000', 'modified': '2024-05-08T20:07:36.993402+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14504}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14504': {'gid': 14504, 'name': 'foo.smithi083.dfkbdq', 'rank': 0, 'incarnation': 3, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.83:6835/486534342', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 486534342}, {'type': 'v1', 'addr': '172.21.15.83:6835', 'nonce': 486534342}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'foo', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-05-08T20:07:42.315 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-08T20:07:42.315 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.foo.smithi083.dfkbdq status 2024-05-08T20:07:42.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:41 smithi188 ceph-mon[48761]: Deploying daemon mds.bar.smithi083.vqxrol on smithi083 2024-05-08T20:07:42.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:41 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/1272564396' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:07:42.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:41 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/1878635949' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-05-08T20:07:42.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:41 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/90474399' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-08T20:07:42.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-05-08T20:07:42.455+0000 7f8bddffb700 0 client.14578 ms_handle_reset on v2:172.21.15.83:6834/486534342 2024-05-08T20:07:42.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-05-08T20:07:42.476+0000 7f8bddffb700 0 client.14582 ms_handle_reset on v2:172.21.15.83:6834/486534342 2024-05-08T20:07:42.468 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-05-08T20:07:42.468 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_fsid": "526038a6-0d75-11ef-bc97-c7b262605968", 2024-05-08T20:07:42.468 INFO:teuthology.orchestra.run.smithi083.stdout: "whoami": 0, 2024-05-08T20:07:42.468 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 14504, 2024-05-08T20:07:42.468 INFO:teuthology.orchestra.run.smithi083.stdout: "want_state": "up:active", 2024-05-08T20:07:42.468 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "up:active", 2024-05-08T20:07:42.469 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "foo", 2024-05-08T20:07:42.469 INFO:teuthology.orchestra.run.smithi083.stdout: "rank_uptime": 9.6962560450000002, 2024-05-08T20:07:42.469 INFO:teuthology.orchestra.run.smithi083.stdout: "mdsmap_epoch": 8, 2024-05-08T20:07:42.469 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch": 69, 2024-05-08T20:07:42.469 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch_barrier": 68, 2024-05-08T20:07:42.469 INFO:teuthology.orchestra.run.smithi083.stdout: "uptime": 10.216939156 2024-05-08T20:07:42.469 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-05-08T20:07:42.480 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T20:07:42.826 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:07:42.826 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":69,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","created":"2024-05-08T19:59:07.152446+0000","modified":"2024-05-08T20:07:37.979695+0000","last_up_change":"2024-05-08T20:05:57.759893+0000","last_in_change":"2024-05-08T20:05:40.402230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T20:03:44.088148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.foo.meta","create_time":"2024-05-08T20:07:27.766436+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.data","create_time":"2024-05-08T20:07:28.151095+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.bar.meta","create_time":"2024-05-08T20:07:34.236025+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bar"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs.bar.data","create_time":"2024-05-08T20:07:35.187905+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bar"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78051d3d-3644-4876-af5f-8c956a8491bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6803","nonce":345419579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6805","nonce":345419579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6809","nonce":345419579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6807","nonce":345419579}]},"public_addr":"172.21.15.83:6803/345419579","cluster_addr":"172.21.15.83:6805/345419579","heartbeat_back_addr":"172.21.15.83:6809/345419579","heartbeat_front_addr":"172.21.15.83:6807/345419579","state":["exists","up"]},{"osd":1,"uuid":"bc62b8e9-e30e-41f1-aaee-4c4e356be487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6811","nonce":1992633959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6813","nonce":1992633959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6817","nonce":1992633959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6815","nonce":1992633959}]},"public_addr":"172.21.15.83:6811/1992633959","cluster_addr":"172.21.15.83:6813/1992633959","heartbeat_back_addr":"172.21.15.83:6817/1992633959","heartbeat_front_addr":"172.21.15.83:6815/1992633959","state":["exists","up"]},{"osd":2,"uuid":"4ae0453a-718b-445f-ad00-a3696fec1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6819","nonce":561864588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6821","nonce":561864588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6825","nonce":561864588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6823","nonce":561864588}]},"public_addr":"172.21.15.83:6819/561864588","cluster_addr":"172.21.15.83:6821/561864588","heartbeat_back_addr":"172.21.15.83:6825/561864588","heartbeat_front_addr":"172.21.15.83:6823/561864588","state":["exists","up"]},{"osd":3,"uuid":"2085ec85-187a-4d69-a721-793cfed7d044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6827","nonce":866184273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6829","nonce":866184273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6833","nonce":866184273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6831","nonce":866184273}]},"public_addr":"172.21.15.83:6827/866184273","cluster_addr":"172.21.15.83:6829/866184273","heartbeat_back_addr":"172.21.15.83:6833/866184273","heartbeat_front_addr":"172.21.15.83:6831/866184273","state":["exists","up"]},{"osd":4,"uuid":"b92dad8b-5bd0-4cd1-ba8d-2920927f132e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6801","nonce":422247785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6803","nonce":422247785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6807","nonce":422247785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6805","nonce":422247785}]},"public_addr":"172.21.15.188:6801/422247785","cluster_addr":"172.21.15.188:6803/422247785","heartbeat_back_addr":"172.21.15.188:6807/422247785","heartbeat_front_addr":"172.21.15.188:6805/422247785","state":["exists","up"]},{"osd":5,"uuid":"aeeda050-668f-47a2-90d0-32e1aa35bfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6809","nonce":841652631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6811","nonce":841652631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6815","nonce":841652631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6813","nonce":841652631}]},"public_addr":"172.21.15.188:6809/841652631","cluster_addr":"172.21.15.188:6811/841652631","heartbeat_back_addr":"172.21.15.188:6815/841652631","heartbeat_front_addr":"172.21.15.188:6813/841652631","state":["exists","up"]},{"osd":6,"uuid":"d3720c87-ec2b-490a-937b-a29833f0267c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6817","nonce":118382910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6819","nonce":118382910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6823","nonce":118382910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6821","nonce":118382910}]},"public_addr":"172.21.15.188:6817/118382910","cluster_addr":"172.21.15.188:6819/118382910","heartbeat_back_addr":"172.21.15.188:6823/118382910","heartbeat_front_addr":"172.21.15.188:6821/118382910","state":["exists","up"]},{"osd":7,"uuid":"8fffd2a6-163d-4a9b-adc2-25329a3803ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6825","nonce":3349446159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6827","nonce":3349446159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6831","nonce":3349446159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6829","nonce":3349446159}]},"public_addr":"172.21.15.188:6825/3349446159","cluster_addr":"172.21.15.188:6827/3349446159","heartbeat_back_addr":"172.21.15.188:6831/3349446159","heartbeat_front_addr":"172.21.15.188:6829/3349446159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:02:42.828169+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:11.944380+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:41.150631+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:11.648151+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:36.764713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:03.134028+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:29.964086+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:56.788654+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:6801/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3612788753":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3117466271":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/3109058932":"2024-05-09T20:01:23.760420+0000","172.21.15.83:6800/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6800/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/890224504":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/2142540104":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/1765857891":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6801/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6800/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/2981330630":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/2443488799":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6801/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/3512048858":"2024-05-09T19:59:36.209291+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T20:07:42.842 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T20:03:44.088148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.foo.meta', 'create_time': '2024-05-08T20:07:27.766436+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.foo.data', 'create_time': '2024-05-08T20:07:28.151095+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs.bar.meta', 'create_time': '2024-05-08T20:07:34.236025+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'bar'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs.bar.data', 'create_time': '2024-05-08T20:07:35.187905+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'bar'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T20:07:42.842 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T20:07:43.207 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-05-08T20:07:43.222 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.meta pg_num 2024-05-08T20:07:43.249 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:42 smithi083 ceph-mon[38695]: pgmap v246: 129 pgs: 3 creating+peering, 126 active+clean; 581 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s wr, 9 op/s 2024-05-08T20:07:43.249 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:42 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/919388170' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:07:43.249 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:42 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3783416971' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:07:43.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:42 smithi188 ceph-mon[48761]: pgmap v246: 129 pgs: 3 creating+peering, 126 active+clean; 581 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s wr, 9 op/s 2024-05-08T20:07:43.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:42 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/919388170' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:07:43.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:42 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/3783416971' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:07:43.560 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:07:43.574 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.data pg_num 2024-05-08T20:07:43.961 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:07:43.976 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.meta pg_num 2024-05-08T20:07:44.316 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:07:44.329 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:44 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/1073830262' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-08T20:07:44.329 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:44 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/1607840823' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:07:44.329 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:44 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:44.329 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:44 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:44.329 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:44 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:44.329 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:44 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.bar.smithi188.uvkoxd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-08T20:07:44.329 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:44 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.bar.smithi188.uvkoxd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-08T20:07:44.330 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:44 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:07:44.330 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:44 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3384516019' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-05-08T20:07:44.331 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.data pg_num 2024-05-08T20:07:44.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:44 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/1073830262' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-08T20:07:44.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:44 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/1607840823' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:07:44.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:44 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:44.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:44 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:44.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:44 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:44.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:44 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.bar.smithi188.uvkoxd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-08T20:07:44.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:44 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.bar.smithi188.uvkoxd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-08T20:07:44.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:44 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:07:44.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:44 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/3384516019' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-05-08T20:07:44.657 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:07:44.671 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T20:07:45.046 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:07:45.046 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24341,"name":"foo.smithi188.tcwiuw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/2884290862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":2884290862},{"type":"v1","addr":"172.21.15.188:6833","nonce":2884290862}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T20:07:28.926384+0000","modified":"2024-05-08T20:07:36.993402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14504},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14504":{"gid":14504,"name":"foo.smithi083.dfkbdq","rank":0,"incarnation":3,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/486534342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":486534342},{"type":"v1","addr":"172.21.15.83:6835","nonce":486534342}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"foo","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1},{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T20:07:35.967161+0000","modified":"2024-05-08T20:07:45.025544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14598":{"gid":14598,"name":"bar.smithi083.vqxrol","rank":0,"incarnation":10,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.83:6837/198299568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":198299568},{"type":"v1","addr":"172.21.15.83:6837","nonce":198299568}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"bar","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":2}]} 2024-05-08T20:07:45.049 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 11 2024-05-08T20:07:45.061 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T20:07:45.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:45 smithi188 ceph-mon[48761]: Deploying daemon mds.bar.smithi188.uvkoxd on smithi188 2024-05-08T20:07:45.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:45 smithi188 ceph-mon[48761]: pgmap v247: 129 pgs: 129 active+clean; 581 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s wr, 6 op/s 2024-05-08T20:07:45.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:45 smithi188 ceph-mon[48761]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-05-08T20:07:45.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:45 smithi188 ceph-mon[48761]: Cluster is now healthy 2024-05-08T20:07:45.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:45 smithi188 ceph-mon[48761]: mds.? [v2:172.21.15.83:6836/198299568,v1:172.21.15.83:6837/198299568] up:boot 2024-05-08T20:07:45.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:45 smithi188 ceph-mon[48761]: Dropping low affinity active daemon mds.foo.smithi188.tcwiuw in favor of higher affinity standby. 2024-05-08T20:07:45.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:45 smithi188 ceph-mon[48761]: Replacing daemon mds.foo.smithi188.tcwiuw as rank 0 with standby daemon mds.bar.smithi083.vqxrol 2024-05-08T20:07:45.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:45 smithi188 ceph-mon[48761]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-08T20:07:45.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:45 smithi188 ceph-mon[48761]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-05-08T20:07:45.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:45 smithi188 ceph-mon[48761]: fsmap foo:1 bar:1 {bar:0=foo.smithi188.tcwiuw=up:active,foo:0=foo.smithi083.dfkbdq=up:active} 1 up:standby 2024-05-08T20:07:45.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:45 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mds metadata", "who": "bar.smithi083.vqxrol"}]: dispatch 2024-05-08T20:07:45.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:45 smithi188 ceph-mon[48761]: osdmap e70: 8 total, 8 up, 8 in 2024-05-08T20:07:45.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:45 smithi188 ceph-mon[48761]: fsmap foo:1 bar:1/1 {bar:0=bar.smithi083.vqxrol=up:replay,foo:0=foo.smithi083.dfkbdq=up:active} 2024-05-08T20:07:45.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:45 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/4043992577' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:07:45.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:45 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/1986780000' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-05-08T20:07:45.387 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:07:45.387 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":70,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","created":"2024-05-08T19:59:07.152446+0000","modified":"2024-05-08T20:07:44.002359+0000","last_up_change":"2024-05-08T20:05:57.759893+0000","last_in_change":"2024-05-08T20:05:40.402230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T20:03:44.088148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.foo.meta","create_time":"2024-05-08T20:07:27.766436+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.data","create_time":"2024-05-08T20:07:28.151095+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.bar.meta","create_time":"2024-05-08T20:07:34.236025+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bar"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs.bar.data","create_time":"2024-05-08T20:07:35.187905+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bar"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78051d3d-3644-4876-af5f-8c956a8491bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6803","nonce":345419579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6805","nonce":345419579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6809","nonce":345419579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6807","nonce":345419579}]},"public_addr":"172.21.15.83:6803/345419579","cluster_addr":"172.21.15.83:6805/345419579","heartbeat_back_addr":"172.21.15.83:6809/345419579","heartbeat_front_addr":"172.21.15.83:6807/345419579","state":["exists","up"]},{"osd":1,"uuid":"bc62b8e9-e30e-41f1-aaee-4c4e356be487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6811","nonce":1992633959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6813","nonce":1992633959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6817","nonce":1992633959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6815","nonce":1992633959}]},"public_addr":"172.21.15.83:6811/1992633959","cluster_addr":"172.21.15.83:6813/1992633959","heartbeat_back_addr":"172.21.15.83:6817/1992633959","heartbeat_front_addr":"172.21.15.83:6815/1992633959","state":["exists","up"]},{"osd":2,"uuid":"4ae0453a-718b-445f-ad00-a3696fec1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6819","nonce":561864588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6821","nonce":561864588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6825","nonce":561864588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6823","nonce":561864588}]},"public_addr":"172.21.15.83:6819/561864588","cluster_addr":"172.21.15.83:6821/561864588","heartbeat_back_addr":"172.21.15.83:6825/561864588","heartbeat_front_addr":"172.21.15.83:6823/561864588","state":["exists","up"]},{"osd":3,"uuid":"2085ec85-187a-4d69-a721-793cfed7d044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6827","nonce":866184273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6829","nonce":866184273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6833","nonce":866184273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6831","nonce":866184273}]},"public_addr":"172.21.15.83:6827/866184273","cluster_addr":"172.21.15.83:6829/866184273","heartbeat_back_addr":"172.21.15.83:6833/866184273","heartbeat_front_addr":"172.21.15.83:6831/866184273","state":["exists","up"]},{"osd":4,"uuid":"b92dad8b-5bd0-4cd1-ba8d-2920927f132e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6801","nonce":422247785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6803","nonce":422247785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6807","nonce":422247785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6805","nonce":422247785}]},"public_addr":"172.21.15.188:6801/422247785","cluster_addr":"172.21.15.188:6803/422247785","heartbeat_back_addr":"172.21.15.188:6807/422247785","heartbeat_front_addr":"172.21.15.188:6805/422247785","state":["exists","up"]},{"osd":5,"uuid":"aeeda050-668f-47a2-90d0-32e1aa35bfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6809","nonce":841652631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6811","nonce":841652631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6815","nonce":841652631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6813","nonce":841652631}]},"public_addr":"172.21.15.188:6809/841652631","cluster_addr":"172.21.15.188:6811/841652631","heartbeat_back_addr":"172.21.15.188:6815/841652631","heartbeat_front_addr":"172.21.15.188:6813/841652631","state":["exists","up"]},{"osd":6,"uuid":"d3720c87-ec2b-490a-937b-a29833f0267c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6817","nonce":118382910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6819","nonce":118382910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6823","nonce":118382910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6821","nonce":118382910}]},"public_addr":"172.21.15.188:6817/118382910","cluster_addr":"172.21.15.188:6819/118382910","heartbeat_back_addr":"172.21.15.188:6823/118382910","heartbeat_front_addr":"172.21.15.188:6821/118382910","state":["exists","up"]},{"osd":7,"uuid":"8fffd2a6-163d-4a9b-adc2-25329a3803ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6825","nonce":3349446159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6827","nonce":3349446159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6831","nonce":3349446159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6829","nonce":3349446159}]},"public_addr":"172.21.15.188:6825/3349446159","cluster_addr":"172.21.15.188:6827/3349446159","heartbeat_back_addr":"172.21.15.188:6831/3349446159","heartbeat_front_addr":"172.21.15.188:6829/3349446159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:02:42.828169+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:11.944380+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:41.150631+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:11.648151+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:36.764713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:03.134028+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:29.964086+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:56.788654+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.188:6833/1664076600":"2024-05-09T20:07:44.002323+0000","172.21.15.83:0/3512048858":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6800/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/2443488799":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6801/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/2142540104":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/890224504":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6801/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/2981330630":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6800/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6800/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3109058932":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3117466271":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/1765857891":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/3612788753":"2024-05-09T20:01:23.760420+0000","172.21.15.83:6801/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.188:6832/1664076600":"2024-05-09T20:07:44.002323+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T20:07:45.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:45 smithi083 ceph-mon[38695]: Deploying daemon mds.bar.smithi188.uvkoxd on smithi188 2024-05-08T20:07:45.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:45 smithi083 ceph-mon[38695]: pgmap v247: 129 pgs: 129 active+clean; 581 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s wr, 6 op/s 2024-05-08T20:07:45.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:45 smithi083 ceph-mon[38695]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-05-08T20:07:45.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:45 smithi083 ceph-mon[38695]: Cluster is now healthy 2024-05-08T20:07:45.400 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:45 smithi083 ceph-mon[38695]: mds.? [v2:172.21.15.83:6836/198299568,v1:172.21.15.83:6837/198299568] up:boot 2024-05-08T20:07:45.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:45 smithi083 ceph-mon[38695]: Dropping low affinity active daemon mds.foo.smithi188.tcwiuw in favor of higher affinity standby. 2024-05-08T20:07:45.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:45 smithi083 ceph-mon[38695]: Replacing daemon mds.foo.smithi188.tcwiuw as rank 0 with standby daemon mds.bar.smithi083.vqxrol 2024-05-08T20:07:45.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:45 smithi083 ceph-mon[38695]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-08T20:07:45.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:45 smithi083 ceph-mon[38695]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-05-08T20:07:45.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:45 smithi083 ceph-mon[38695]: fsmap foo:1 bar:1 {bar:0=foo.smithi188.tcwiuw=up:active,foo:0=foo.smithi083.dfkbdq=up:active} 1 up:standby 2024-05-08T20:07:45.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:45 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mds metadata", "who": "bar.smithi083.vqxrol"}]: dispatch 2024-05-08T20:07:45.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:45 smithi083 ceph-mon[38695]: osdmap e70: 8 total, 8 up, 8 in 2024-05-08T20:07:45.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:45 smithi083 ceph-mon[38695]: fsmap foo:1 bar:1/1 {bar:0=bar.smithi083.vqxrol=up:replay,foo:0=foo.smithi083.dfkbdq=up:active} 2024-05-08T20:07:45.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:45 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/4043992577' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:07:45.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:45 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/1986780000' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-05-08T20:07:45.403 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-08T20:07:35.967161+0000', 'modified': '2024-05-08T20:07:36.993402+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24273}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24273': {'gid': 24273, 'name': 'foo.smithi188.tcwiuw', 'rank': 0, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.188:6833/1664076600', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6832', 'nonce': 1664076600}, {'type': 'v1', 'addr': '172.21.15.188:6833', 'nonce': 1664076600}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'bar', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-05-08T20:07:45.403 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-08T20:07:45.403 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.foo.smithi188.tcwiuw status 2024-05-08T20:07:45.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-05-08T20:07:45.544+0000 7f37f4ff9700 0 client.14630 ms_handle_reset on v2:172.21.15.188:6832/2884290862 2024-05-08T20:07:45.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-05-08T20:07:45.565+0000 7f37f4ff9700 0 client.14634 ms_handle_reset on v2:172.21.15.188:6832/2884290862 2024-05-08T20:07:45.556 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-05-08T20:07:45.556 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_fsid": "526038a6-0d75-11ef-bc97-c7b262605968", 2024-05-08T20:07:45.556 INFO:teuthology.orchestra.run.smithi083.stdout: "whoami": -1, 2024-05-08T20:07:45.556 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 24341, 2024-05-08T20:07:45.556 INFO:teuthology.orchestra.run.smithi083.stdout: "want_state": "up:standby", 2024-05-08T20:07:45.556 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "up:standby", 2024-05-08T20:07:45.556 INFO:teuthology.orchestra.run.smithi083.stdout: "mdsmap_epoch": 11, 2024-05-08T20:07:45.556 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch": 0, 2024-05-08T20:07:45.556 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch_barrier": 0, 2024-05-08T20:07:45.556 INFO:teuthology.orchestra.run.smithi083.stdout: "uptime": 1.5100252860000001 2024-05-08T20:07:45.556 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-05-08T20:07:46.302 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:46 smithi188 ceph-mon[48761]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-05-08T20:07:46.303 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:46 smithi188 ceph-mon[48761]: mds.? [v2:172.21.15.83:6836/198299568,v1:172.21.15.83:6837/198299568] up:reconnect 2024-05-08T20:07:46.303 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:46 smithi188 ceph-mon[48761]: mds.? [v2:172.21.15.188:6832/2884290862,v1:172.21.15.188:6833/2884290862] up:boot 2024-05-08T20:07:46.303 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:46 smithi188 ceph-mon[48761]: fsmap foo:1 bar:1/1 {bar:0=bar.smithi083.vqxrol=up:reconnect,foo:0=foo.smithi083.dfkbdq=up:active} 1 up:standby 2024-05-08T20:07:46.303 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:46 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mds metadata", "who": "foo.smithi188.tcwiuw"}]: dispatch 2024-05-08T20:07:46.303 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:46 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/686496988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-08T20:07:46.303 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:46 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/3383614631' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:07:46.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:46 smithi083 ceph-mon[38695]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-05-08T20:07:46.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:46 smithi083 ceph-mon[38695]: mds.? [v2:172.21.15.83:6836/198299568,v1:172.21.15.83:6837/198299568] up:reconnect 2024-05-08T20:07:46.401 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:46 smithi083 ceph-mon[38695]: mds.? [v2:172.21.15.188:6832/2884290862,v1:172.21.15.188:6833/2884290862] up:boot 2024-05-08T20:07:46.402 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:46 smithi083 ceph-mon[38695]: fsmap foo:1 bar:1/1 {bar:0=bar.smithi083.vqxrol=up:reconnect,foo:0=foo.smithi083.dfkbdq=up:active} 1 up:standby 2024-05-08T20:07:46.402 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:46 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mds metadata", "who": "foo.smithi188.tcwiuw"}]: dispatch 2024-05-08T20:07:46.402 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:46 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/686496988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-08T20:07:46.402 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:46 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3383614631' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:07:46.569 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T20:07:46.965 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:07:46.965 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24341,"name":"foo.smithi188.tcwiuw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/2884290862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":2884290862},{"type":"v1","addr":"172.21.15.188:6833","nonce":2884290862}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T20:07:28.926384+0000","modified":"2024-05-08T20:07:36.993402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14504},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14504":{"gid":14504,"name":"foo.smithi083.dfkbdq","rank":0,"incarnation":3,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/486534342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":486534342},{"type":"v1","addr":"172.21.15.83:6835","nonce":486534342}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"foo","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1},{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T20:07:35.967161+0000","modified":"2024-05-08T20:07:46.026830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14598":{"gid":14598,"name":"bar.smithi083.vqxrol","rank":0,"incarnation":10,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.83:6837/198299568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":198299568},{"type":"v1","addr":"172.21.15.83:6837","nonce":198299568}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"bar","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":2}]} 2024-05-08T20:07:46.968 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 12 2024-05-08T20:07:46.981 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 12, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-08T20:07:35.967161+0000', 'modified': '2024-05-08T20:07:46.026830+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 70, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14598}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14598': {'gid': 14598, 'name': 'bar.smithi083.vqxrol', 'rank': 0, 'incarnation': 10, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.83:6837/198299568', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6836', 'nonce': 198299568}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 198299568}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'bar', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-05-08T20:07:46.981 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14598:up:rejoin 2024-05-08T20:07:47.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:47 smithi188 ceph-mon[48761]: pgmap v249: 129 pgs: 129 active+clean; 581 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-05-08T20:07:47.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:47 smithi188 ceph-mon[48761]: mds.? [v2:172.21.15.83:6836/198299568,v1:172.21.15.83:6837/198299568] up:rejoin 2024-05-08T20:07:47.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:47 smithi188 ceph-mon[48761]: fsmap foo:1 bar:1/1 {bar:0=bar.smithi083.vqxrol=up:rejoin,foo:0=foo.smithi083.dfkbdq=up:active} 1 up:standby 2024-05-08T20:07:47.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:47 smithi188 ceph-mon[48761]: daemon mds.bar.smithi083.vqxrol is now active in filesystem bar as rank 0 2024-05-08T20:07:47.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:47 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:47.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:47 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:47.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:47 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:47.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:47 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:47.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:47 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:47.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:47 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:07:47.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:47 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/2372515759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-08T20:07:47.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:47 smithi083 ceph-mon[38695]: pgmap v249: 129 pgs: 129 active+clean; 581 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-05-08T20:07:47.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:47 smithi083 ceph-mon[38695]: mds.? [v2:172.21.15.83:6836/198299568,v1:172.21.15.83:6837/198299568] up:rejoin 2024-05-08T20:07:47.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:47 smithi083 ceph-mon[38695]: fsmap foo:1 bar:1/1 {bar:0=bar.smithi083.vqxrol=up:rejoin,foo:0=foo.smithi083.dfkbdq=up:active} 1 up:standby 2024-05-08T20:07:47.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:47 smithi083 ceph-mon[38695]: daemon mds.bar.smithi083.vqxrol is now active in filesystem bar as rank 0 2024-05-08T20:07:47.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:47 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:47.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:47 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:47.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:47 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:47.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:47 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:47.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:47 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:47.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:47 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:07:47.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:47 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2372515759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-08T20:07:47.982 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-08T20:07:48.289 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:48 smithi188 ceph-mon[48761]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-08T20:07:48.289 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:48 smithi188 ceph-mon[48761]: Cluster is now healthy 2024-05-08T20:07:48.289 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:48 smithi188 ceph-mon[48761]: mds.? [v2:172.21.15.83:6836/198299568,v1:172.21.15.83:6837/198299568] up:active 2024-05-08T20:07:48.289 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:48 smithi188 ceph-mon[48761]: mds.? [v2:172.21.15.188:6834/3571797326,v1:172.21.15.188:6835/3571797326] up:boot 2024-05-08T20:07:48.289 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:48 smithi188 ceph-mon[48761]: fsmap foo:1 bar:1 {bar:0=bar.smithi083.vqxrol=up:active,foo:0=foo.smithi083.dfkbdq=up:active} 2 up:standby 2024-05-08T20:07:48.289 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:48 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mds metadata", "who": "bar.smithi188.uvkoxd"}]: dispatch 2024-05-08T20:07:48.351 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:48 smithi083 ceph-mon[38695]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-08T20:07:48.351 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:48 smithi083 ceph-mon[38695]: Cluster is now healthy 2024-05-08T20:07:48.351 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:48 smithi083 ceph-mon[38695]: mds.? [v2:172.21.15.83:6836/198299568,v1:172.21.15.83:6837/198299568] up:active 2024-05-08T20:07:48.351 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:48 smithi083 ceph-mon[38695]: mds.? [v2:172.21.15.188:6834/3571797326,v1:172.21.15.188:6835/3571797326] up:boot 2024-05-08T20:07:48.351 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:48 smithi083 ceph-mon[38695]: fsmap foo:1 bar:1 {bar:0=bar.smithi083.vqxrol=up:active,foo:0=foo.smithi083.dfkbdq=up:active} 2 up:standby 2024-05-08T20:07:48.351 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:48 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "mds metadata", "who": "bar.smithi188.uvkoxd"}]: dispatch 2024-05-08T20:07:48.375 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:07:48.375 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24341,"name":"foo.smithi188.tcwiuw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/2884290862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":2884290862},{"type":"v1","addr":"172.21.15.188:6833","nonce":2884290862}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24363,"name":"bar.smithi188.uvkoxd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/3571797326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":3571797326},{"type":"v1","addr":"172.21.15.188:6835","nonce":3571797326}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T20:07:28.926384+0000","modified":"2024-05-08T20:07:36.993402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14504},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14504":{"gid":14504,"name":"foo.smithi083.dfkbdq","rank":0,"incarnation":3,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/486534342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":486534342},{"type":"v1","addr":"172.21.15.83:6835","nonce":486534342}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"foo","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1},{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-08T20:07:35.967161+0000","modified":"2024-05-08T20:07:47.032999+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14598":{"gid":14598,"name":"bar.smithi083.vqxrol","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/198299568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":198299568},{"type":"v1","addr":"172.21.15.83:6837","nonce":198299568}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"bar","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":2}]} 2024-05-08T20:07:48.377 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 13 2024-05-08T20:07:48.389 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 13, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-08T20:07:35.967161+0000', 'modified': '2024-05-08T20:07:47.032999+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 70, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14598}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14598': {'gid': 14598, 'name': 'bar.smithi083.vqxrol', 'rank': 0, 'incarnation': 10, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/198299568', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6836', 'nonce': 198299568}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 198299568}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'bar', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-05-08T20:07:48.390 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-08T20:07:48.390 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.bar.smithi083.vqxrol status 2024-05-08T20:07:48.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-05-08T20:07:48.533+0000 7ff6baff5700 0 client.14650 ms_handle_reset on v2:172.21.15.83:6836/198299568 2024-05-08T20:07:48.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-05-08T20:07:48.554+0000 7ff6baff5700 0 client.14654 ms_handle_reset on v2:172.21.15.83:6836/198299568 2024-05-08T20:07:48.545 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-05-08T20:07:48.545 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_fsid": "526038a6-0d75-11ef-bc97-c7b262605968", 2024-05-08T20:07:48.545 INFO:teuthology.orchestra.run.smithi083.stdout: "whoami": 0, 2024-05-08T20:07:48.545 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 14598, 2024-05-08T20:07:48.546 INFO:teuthology.orchestra.run.smithi083.stdout: "want_state": "up:active", 2024-05-08T20:07:48.546 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "up:active", 2024-05-08T20:07:48.546 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "bar", 2024-05-08T20:07:48.546 INFO:teuthology.orchestra.run.smithi083.stdout: "rank_uptime": 4.5289265009999999, 2024-05-08T20:07:48.546 INFO:teuthology.orchestra.run.smithi083.stdout: "mdsmap_epoch": 13, 2024-05-08T20:07:48.546 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch": 70, 2024-05-08T20:07:48.546 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch_barrier": 70, 2024-05-08T20:07:48.546 INFO:teuthology.orchestra.run.smithi083.stdout: "uptime": 4.8811206650000001 2024-05-08T20:07:48.546 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-05-08T20:07:48.557 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-05-08T20:07:48.582 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-05-08T20:07:48.583 INFO:tasks.ceph_fuse:config is {'client.0': {'cephfs_name': 'foo'}} 2024-05-08T20:07:48.583 INFO:tasks.ceph_fuse:client.0 config is {'cephfs_name': 'foo'} 2024-05-08T20:07:48.583 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-08T20:07:48.583 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-08T20:07:48.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:07:48.583 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list 2024-05-08T20:07:48.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:07:48.601 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link delete ceph-brx 2024-05-08T20:07:48.671 INFO:teuthology.orchestra.run.smithi083.stderr:Cannot find device "ceph-brx" 2024-05-08T20:07:48.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T20:07:48.673 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-05-08T20:07:48.673 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph fs ls 2024-05-08T20:07:49.314 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:07:49.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:49 smithi188 ceph-mon[48761]: pgmap v250: 129 pgs: 129 active+clean; 581 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-05-08T20:07:49.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:49 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/3042403939' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-08T20:07:49.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:49 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:49.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:49 smithi083 ceph-mon[38695]: pgmap v250: 129 pgs: 129 active+clean; 581 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-05-08T20:07:49.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:49 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3042403939' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-08T20:07:49.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:49 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:50.438 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:50 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:50.438 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:50 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:50.439 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:50 smithi083 ceph-mon[38695]: pgmap v251: 129 pgs: 129 active+clean; 581 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 102 B/s wr, 7 op/s 2024-05-08T20:07:50.584 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:50 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:50.585 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:50 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:50.585 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:50 smithi188 ceph-mon[48761]: pgmap v251: 129 pgs: 129 active+clean; 581 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 102 B/s wr, 7 op/s 2024-05-08T20:07:51.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:51 smithi083 ceph-mon[38695]: mds.? [v2:172.21.15.83:6836/198299568,v1:172.21.15.83:6837/198299568] up:active 2024-05-08T20:07:51.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:51 smithi083 ceph-mon[38695]: fsmap foo:1 bar:1 {bar:0=bar.smithi083.vqxrol=up:active,foo:0=foo.smithi083.dfkbdq=up:active} 2 up:standby 2024-05-08T20:07:51.584 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:51 smithi188 ceph-mon[48761]: mds.? [v2:172.21.15.83:6836/198299568,v1:172.21.15.83:6837/198299568] up:active 2024-05-08T20:07:51.585 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:51 smithi188 ceph-mon[48761]: fsmap foo:1 bar:1 {bar:0=bar.smithi083.vqxrol=up:active,foo:0=foo.smithi083.dfkbdq=up:active} 2 up:standby 2024-05-08T20:07:51.825 INFO:teuthology.orchestra.run.smithi083.stdout:name: foo, metadata pool: cephfs.foo.meta, data pools: [cephfs.foo.data ] 2024-05-08T20:07:51.825 INFO:teuthology.orchestra.run.smithi083.stdout:name: bar, metadata pool: cephfs.bar.meta, data pools: [cephfs.bar.data ] 2024-05-08T20:07:52.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:52 smithi083 ceph-mon[38695]: mds.? [v2:172.21.15.188:6834/3571797326,v1:172.21.15.188:6835/3571797326] up:standby 2024-05-08T20:07:52.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:52 smithi083 ceph-mon[38695]: fsmap foo:1 bar:1 {bar:0=bar.smithi083.vqxrol=up:active,foo:0=foo.smithi083.dfkbdq=up:active} 2 up:standby 2024-05-08T20:07:52.434 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:52 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3188286416' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-08T20:07:52.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:52 smithi083 ceph-mon[38695]: pgmap v252: 129 pgs: 129 active+clean; 581 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 7.9 KiB/s rd, 0 B/s wr, 7 op/s 2024-05-08T20:07:52.585 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:52 smithi188 ceph-mon[48761]: mds.? [v2:172.21.15.188:6834/3571797326,v1:172.21.15.188:6835/3571797326] up:standby 2024-05-08T20:07:52.585 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:52 smithi188 ceph-mon[48761]: fsmap foo:1 bar:1 {bar:0=bar.smithi083.vqxrol=up:active,foo:0=foo.smithi083.dfkbdq=up:active} 2 up:standby 2024-05-08T20:07:52.585 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:52 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/3188286416' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-08T20:07:52.585 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:52 smithi188 ceph-mon[48761]: pgmap v252: 129 pgs: 129 active+clean; 581 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 7.9 KiB/s rd, 0 B/s wr, 7 op/s 2024-05-08T20:07:52.634 INFO:tasks.cephfs.mount:Mounting Ceph FS foo; just confirmed its presence on cluster 2024-05-08T20:07:52.634 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-08T20:07:52.634 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi083.front.sepia.ceph.com 2024-05-08T20:07:52.634 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-08T20:07:52.634 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-08T20:07:52.634 INFO:tasks.cephfs.mount:self.cephfs_name = foo 2024-05-08T20:07:52.634 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-08T20:07:52.634 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-08T20:07:52.634 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-08T20:07:52.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:07:52.635 DEBUG:teuthology.orchestra.run.smithi083:> ip addr 2024-05-08T20:07:52.655 INFO:teuthology.orchestra.run.smithi083.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-08T20:07:52.655 INFO:teuthology.orchestra.run.smithi083.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-08T20:07:52.655 INFO:teuthology.orchestra.run.smithi083.stdout: inet 127.0.0.1/8 scope host lo 2024-05-08T20:07:52.655 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-05-08T20:07:52.655 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 ::1/128 scope host 2024-05-08T20:07:52.655 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-05-08T20:07:52.655 INFO:teuthology.orchestra.run.smithi083.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T20:07:52.655 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1e brd ff:ff:ff:ff:ff:ff 2024-05-08T20:07:52.655 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f0 2024-05-08T20:07:52.655 INFO:teuthology.orchestra.run.smithi083.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T20:07:52.655 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1f brd ff:ff:ff:ff:ff:ff 2024-05-08T20:07:52.655 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f1 2024-05-08T20:07:52.656 INFO:teuthology.orchestra.run.smithi083.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T20:07:52.656 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:90 brd ff:ff:ff:ff:ff:ff 2024-05-08T20:07:52.656 INFO:teuthology.orchestra.run.smithi083.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-08T20:07:52.656 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:91 brd ff:ff:ff:ff:ff:ff 2024-05-08T20:07:52.656 INFO:teuthology.orchestra.run.smithi083.stdout: inet 172.21.15.83/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-05-08T20:07:52.656 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft 42011sec preferred_lft 42011sec 2024-05-08T20:07:52.656 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::ec4:7aff:fe8f:ce91/64 scope link 2024-05-08T20:07:52.656 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-05-08T20:07:52.656 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-08T20:07:52.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:07:52.657 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-05-08T20:07:52.657 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-05-08T20:07:52.657 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link add name ceph-brx type bridge 2024-05-08T20:07:52.657 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip addr flush dev ceph-brx 2024-05-08T20:07:52.657 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link set ceph-brx up 2024-05-08T20:07:52.657 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-08T20:07:52.657 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-05-08T20:07:52.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:07:52.865 DEBUG:teuthology.orchestra.run.smithi083:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-08T20:07:52.893 INFO:teuthology.orchestra.run.smithi083.stdout:1 2024-05-08T20:07:52.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:07:52.894 DEBUG:teuthology.orchestra.run.smithi083:> route 2024-05-08T20:07:53.006 INFO:teuthology.orchestra.run.smithi083.stdout:Kernel IP routing table 2024-05-08T20:07:53.006 INFO:teuthology.orchestra.run.smithi083.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-08T20:07:53.006 INFO:teuthology.orchestra.run.smithi083.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-08T20:07:53.006 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-08T20:07:53.006 INFO:teuthology.orchestra.run.smithi083.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-08T20:07:53.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:07:53.007 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-05-08T20:07:53.007 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-05-08T20:07:53.007 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-08T20:07:53.007 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-08T20:07:53.007 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-08T20:07:53.007 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-05-08T20:07:53.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:07:53.492 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list 2024-05-08T20:07:53.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:07:53.510 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list-id 2024-05-08T20:07:53.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:07:53.567 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-05-08T20:07:53.567 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-05-08T20:07:53.567 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-08T20:07:53.567 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-08T20:07:53.567 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-05-08T20:07:53.672 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-08T20:07:53.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:07:53.672 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-05-08T20:07:53.672 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-05-08T20:07:53.672 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-08T20:07:53.672 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-08T20:07:53.672 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-08T20:07:53.672 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-08T20:07:53.673 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-08T20:07:53.673 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-05-08T20:07:53.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:07:53.867 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-05-08T20:07:53.867 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-05-08T20:07:53.867 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link set brx.0 up 2024-05-08T20:07:53.867 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link set dev brx.0 master ceph-brx 2024-05-08T20:07:53.867 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-05-08T20:07:53.967 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'cephfs_name': 'foo'} 2024-05-08T20:07:53.967 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-08T20:07:53.967 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-08T20:07:53.985 INFO:teuthology.orchestra.run.smithi083.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-08T20:07:53.985 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-08T20:07:53.985 DEBUG:teuthology.orchestra.run.smithi083:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-08T20:07:54.042 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-05-08T20:07:54.110 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-05-08T20:07:54.110 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-05-08T20:07:54.177 INFO:teuthology.orchestra.run.smithi083.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-05-08T20:07:54.178 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-05-08T20:07:54.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:07:54.179 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-05-08T20:07:54.237 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-08T20:07:54.237 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_fs=foo) 2024-05-08T20:07:54.280 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-05-08T20:07:54.306 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-05-08T20:07:54.307 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-05-08T20:07:54.372 INFO:teuthology.orchestra.run.smithi083.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-05-08T20:07:54.374 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-05-08T20:07:54.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:07:54.374 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-05-08T20:07:54.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:2024-05-08T20:07:54.399+0000 7fb03283e5c0 -1 init, newargv = 0x55a396a92750 newargc=15 2024-05-08T20:07:54.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[107141]: starting ceph client 2024-05-08T20:07:54.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[107141]: starting fuse 2024-05-08T20:07:54.408 INFO:teuthology.orchestra.run.smithi083.stdout:177 2024-05-08T20:07:54.408 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [177] 2024-05-08T20:07:54.408 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-05-08T20:07:54.409 DEBUG:teuthology.orchestra.run.smithi083:> import glob 2024-05-08T20:07:54.409 DEBUG:teuthology.orchestra.run.smithi083:> import re 2024-05-08T20:07:54.409 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-05-08T20:07:54.409 DEBUG:teuthology.orchestra.run.smithi083:> import subprocess 2024-05-08T20:07:54.409 DEBUG:teuthology.orchestra.run.smithi083:> 2024-05-08T20:07:54.409 DEBUG:teuthology.orchestra.run.smithi083:> def _find_admin_socket(client_name): 2024-05-08T20:07:54.409 DEBUG:teuthology.orchestra.run.smithi083:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-08T20:07:54.409 DEBUG:teuthology.orchestra.run.smithi083:> files = glob.glob(asok_path) 2024-05-08T20:07:54.409 DEBUG:teuthology.orchestra.run.smithi083:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-08T20:07:54.409 DEBUG:teuthology.orchestra.run.smithi083:> 2024-05-08T20:07:54.409 DEBUG:teuthology.orchestra.run.smithi083:> # Given a non-glob path, it better be there 2024-05-08T20:07:54.409 DEBUG:teuthology.orchestra.run.smithi083:> if "*" not in asok_path: 2024-05-08T20:07:54.409 DEBUG:teuthology.orchestra.run.smithi083:> assert(len(files) == 1) 2024-05-08T20:07:54.410 DEBUG:teuthology.orchestra.run.smithi083:> return files[0] 2024-05-08T20:07:54.410 DEBUG:teuthology.orchestra.run.smithi083:> 2024-05-08T20:07:54.410 DEBUG:teuthology.orchestra.run.smithi083:> for f in files: 2024-05-08T20:07:54.410 DEBUG:teuthology.orchestra.run.smithi083:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-08T20:07:54.410 DEBUG:teuthology.orchestra.run.smithi083:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-08T20:07:54.410 DEBUG:teuthology.orchestra.run.smithi083:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-08T20:07:54.410 DEBUG:teuthology.orchestra.run.smithi083:> contents = proc_f.read() 2024-05-08T20:07:54.410 DEBUG:teuthology.orchestra.run.smithi083:> if mountpoint in contents: 2024-05-08T20:07:54.410 DEBUG:teuthology.orchestra.run.smithi083:> return f 2024-05-08T20:07:54.410 DEBUG:teuthology.orchestra.run.smithi083:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-08T20:07:54.410 DEBUG:teuthology.orchestra.run.smithi083:> 2024-05-08T20:07:54.410 DEBUG:teuthology.orchestra.run.smithi083:> print(_find_admin_socket("client.0")) 2024-05-08T20:07:54.410 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-05-08T20:07:54.532 INFO:teuthology.orchestra.run.smithi083.stdout:/var/run/ceph/ceph-client.0.107141.asok 2024-05-08T20:07:54.715 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.107141.asok 2024-05-08T20:07:54.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:07:54.715 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.107141.asok status 2024-05-08T20:07:54.871 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-05-08T20:07:54.872 INFO:teuthology.orchestra.run.smithi083.stdout: "metadata": { 2024-05-08T20:07:54.872 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_sha1": "d21e4f961318b56f0d915caa2490c3711bf13c34", 2024-05-08T20:07:54.872 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_version": "ceph version 18.2.2-1322-gd21e4f96 (d21e4f961318b56f0d915caa2490c3711bf13c34) reef (stable)", 2024-05-08T20:07:54.872 INFO:teuthology.orchestra.run.smithi083.stdout: "entity_id": "0", 2024-05-08T20:07:54.872 INFO:teuthology.orchestra.run.smithi083.stdout: "hostname": "smithi083", 2024-05-08T20:07:54.872 INFO:teuthology.orchestra.run.smithi083.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-08T20:07:54.872 INFO:teuthology.orchestra.run.smithi083.stdout: "pid": "107141", 2024-05-08T20:07:54.872 INFO:teuthology.orchestra.run.smithi083.stdout: "root": "/" 2024-05-08T20:07:54.872 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-05-08T20:07:54.872 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_count": 0, 2024-05-08T20:07:54.872 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_pinned_count": 0, 2024-05-08T20:07:54.873 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 14662, 2024-05-08T20:07:54.873 INFO:teuthology.orchestra.run.smithi083.stdout: "inst": { 2024-05-08T20:07:54.873 INFO:teuthology.orchestra.run.smithi083.stdout: "name": { 2024-05-08T20:07:54.873 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "client", 2024-05-08T20:07:54.873 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 14662 2024-05-08T20:07:54.873 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-05-08T20:07:54.873 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-05-08T20:07:54.873 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-05-08T20:07:54.873 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-05-08T20:07:54.873 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 1739400375 2024-05-08T20:07:54.873 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-05-08T20:07:54.873 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-05-08T20:07:54.873 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-05-08T20:07:54.874 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-05-08T20:07:54.874 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-05-08T20:07:54.874 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 1739400375 2024-05-08T20:07:54.874 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-05-08T20:07:54.874 INFO:teuthology.orchestra.run.smithi083.stdout: "inst_str": "client.14662 192.168.0.1:0/1739400375", 2024-05-08T20:07:54.874 INFO:teuthology.orchestra.run.smithi083.stdout: "addr_str": "192.168.0.1:0/1739400375", 2024-05-08T20:07:54.874 INFO:teuthology.orchestra.run.smithi083.stdout: "inode_count": 1, 2024-05-08T20:07:54.874 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_epoch": 8, 2024-05-08T20:07:54.874 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch": 70, 2024-05-08T20:07:54.874 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch_barrier": 0, 2024-05-08T20:07:54.874 INFO:teuthology.orchestra.run.smithi083.stdout: "blocklisted": false, 2024-05-08T20:07:54.875 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "foo" 2024-05-08T20:07:54.875 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-05-08T20:07:54.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:07:54.882 DEBUG:teuthology.orchestra.run.smithi083:> stat --file-system '--printf=%T 2024-05-08T20:07:54.882 DEBUG:teuthology.orchestra.run.smithi083:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-08T20:07:54.900 INFO:teuthology.orchestra.run.smithi083.stdout:fuseblk 2024-05-08T20:07:54.900 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-08T20:07:54.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:07:54.900 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-08T20:07:54.968 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-05-08T20:07:54.977 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-05-08T20:07:54.977 INFO:tasks.ceph_fuse:config is {'client.1': {'cephfs_name': 'bar'}} 2024-05-08T20:07:54.977 INFO:tasks.ceph_fuse:client.1 config is {'cephfs_name': 'bar'} 2024-05-08T20:07:54.978 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-08T20:07:54.978 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-05-08T20:07:54.978 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-05-08T20:07:54.978 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T20:07:55.028 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-05-08T20:07:55.028 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T20:07:55.087 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T20:07:55.087 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:55.087 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:07:55.087 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:54 smithi188 ceph-mon[48761]: pgmap v253: 129 pgs: 129 active+clean; 581 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-05-08T20:07:55.088 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:55.088 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:55.088 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:07:55.088 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:07:55.088 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:55.088 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:07:55.088 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:07:55.088 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:07:55.088 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:55.150 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:55.150 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:07:55.150 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:54 smithi083 ceph-mon[38695]: pgmap v253: 129 pgs: 129 active+clean; 581 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-05-08T20:07:55.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:55.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:55.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:07:55.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:07:55.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:55.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:07:55.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:07:55.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:07:55.151 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:07:55.456 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:07:55.456 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":70,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","created":"2024-05-08T19:59:07.152446+0000","modified":"2024-05-08T20:07:44.002359+0000","last_up_change":"2024-05-08T20:05:57.759893+0000","last_in_change":"2024-05-08T20:05:40.402230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T20:03:44.088148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.foo.meta","create_time":"2024-05-08T20:07:27.766436+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.data","create_time":"2024-05-08T20:07:28.151095+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.bar.meta","create_time":"2024-05-08T20:07:34.236025+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bar"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs.bar.data","create_time":"2024-05-08T20:07:35.187905+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bar"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78051d3d-3644-4876-af5f-8c956a8491bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6803","nonce":345419579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6805","nonce":345419579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6809","nonce":345419579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6807","nonce":345419579}]},"public_addr":"172.21.15.83:6803/345419579","cluster_addr":"172.21.15.83:6805/345419579","heartbeat_back_addr":"172.21.15.83:6809/345419579","heartbeat_front_addr":"172.21.15.83:6807/345419579","state":["exists","up"]},{"osd":1,"uuid":"bc62b8e9-e30e-41f1-aaee-4c4e356be487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6811","nonce":1992633959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6813","nonce":1992633959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6817","nonce":1992633959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6815","nonce":1992633959}]},"public_addr":"172.21.15.83:6811/1992633959","cluster_addr":"172.21.15.83:6813/1992633959","heartbeat_back_addr":"172.21.15.83:6817/1992633959","heartbeat_front_addr":"172.21.15.83:6815/1992633959","state":["exists","up"]},{"osd":2,"uuid":"4ae0453a-718b-445f-ad00-a3696fec1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6819","nonce":561864588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6821","nonce":561864588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6825","nonce":561864588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6823","nonce":561864588}]},"public_addr":"172.21.15.83:6819/561864588","cluster_addr":"172.21.15.83:6821/561864588","heartbeat_back_addr":"172.21.15.83:6825/561864588","heartbeat_front_addr":"172.21.15.83:6823/561864588","state":["exists","up"]},{"osd":3,"uuid":"2085ec85-187a-4d69-a721-793cfed7d044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6827","nonce":866184273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6829","nonce":866184273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6833","nonce":866184273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6831","nonce":866184273}]},"public_addr":"172.21.15.83:6827/866184273","cluster_addr":"172.21.15.83:6829/866184273","heartbeat_back_addr":"172.21.15.83:6833/866184273","heartbeat_front_addr":"172.21.15.83:6831/866184273","state":["exists","up"]},{"osd":4,"uuid":"b92dad8b-5bd0-4cd1-ba8d-2920927f132e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6801","nonce":422247785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6803","nonce":422247785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6807","nonce":422247785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6805","nonce":422247785}]},"public_addr":"172.21.15.188:6801/422247785","cluster_addr":"172.21.15.188:6803/422247785","heartbeat_back_addr":"172.21.15.188:6807/422247785","heartbeat_front_addr":"172.21.15.188:6805/422247785","state":["exists","up"]},{"osd":5,"uuid":"aeeda050-668f-47a2-90d0-32e1aa35bfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6809","nonce":841652631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6811","nonce":841652631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6815","nonce":841652631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6813","nonce":841652631}]},"public_addr":"172.21.15.188:6809/841652631","cluster_addr":"172.21.15.188:6811/841652631","heartbeat_back_addr":"172.21.15.188:6815/841652631","heartbeat_front_addr":"172.21.15.188:6813/841652631","state":["exists","up"]},{"osd":6,"uuid":"d3720c87-ec2b-490a-937b-a29833f0267c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6817","nonce":118382910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6819","nonce":118382910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6823","nonce":118382910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6821","nonce":118382910}]},"public_addr":"172.21.15.188:6817/118382910","cluster_addr":"172.21.15.188:6819/118382910","heartbeat_back_addr":"172.21.15.188:6823/118382910","heartbeat_front_addr":"172.21.15.188:6821/118382910","state":["exists","up"]},{"osd":7,"uuid":"8fffd2a6-163d-4a9b-adc2-25329a3803ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6825","nonce":3349446159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6827","nonce":3349446159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6831","nonce":3349446159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6829","nonce":3349446159}]},"public_addr":"172.21.15.188:6825/3349446159","cluster_addr":"172.21.15.188:6827/3349446159","heartbeat_back_addr":"172.21.15.188:6831/3349446159","heartbeat_front_addr":"172.21.15.188:6829/3349446159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:02:42.828169+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:11.944380+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:41.150631+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:11.648151+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:36.764713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:03.134028+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:29.964086+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:56.788654+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.188:6833/1664076600":"2024-05-09T20:07:44.002323+0000","172.21.15.83:0/3512048858":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6800/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/2443488799":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6801/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/2142540104":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/890224504":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6801/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/2981330630":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6800/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6800/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3109058932":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3117466271":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/1765857891":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/3612788753":"2024-05-09T20:01:23.760420+0000","172.21.15.83:6801/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.188:6832/1664076600":"2024-05-09T20:07:44.002323+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T20:07:55.470 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T20:03:44.088148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.foo.meta', 'create_time': '2024-05-08T20:07:27.766436+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.foo.data', 'create_time': '2024-05-08T20:07:28.151095+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs.bar.meta', 'create_time': '2024-05-08T20:07:34.236025+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'bar'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs.bar.data', 'create_time': '2024-05-08T20:07:35.187905+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'bar'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T20:07:55.470 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T20:07:55.837 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-05-08T20:07:55.851 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.meta pg_num 2024-05-08T20:07:56.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:55 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/591966787' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:07:56.176 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:07:56.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:55 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/591966787' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:07:56.190 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.data pg_num 2024-05-08T20:07:56.516 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:07:56.530 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.meta pg_num 2024-05-08T20:07:56.860 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:07:56.874 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.data pg_num 2024-05-08T20:07:57.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:56 smithi188 ceph-mon[48761]: pgmap v254: 129 pgs: 129 active+clean; 581 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 6.9 KiB/s rd, 0 B/s wr, 6 op/s 2024-05-08T20:07:57.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:56 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/2519322938' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-08T20:07:57.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:56 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/2515593758' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:07:57.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:56 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/3646506502' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-05-08T20:07:57.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:56 smithi083 ceph-mon[38695]: pgmap v254: 129 pgs: 129 active+clean; 581 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 6.9 KiB/s rd, 0 B/s wr, 6 op/s 2024-05-08T20:07:57.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:56 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2519322938' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-08T20:07:57.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:56 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2515593758' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:07:57.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:56 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3646506502' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-05-08T20:07:57.200 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:07:57.215 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-08T20:07:57.553 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.188:6833/1664076600 2024-05-09T20:07:44.002323+0000 2024-05-08T20:07:57.553 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/3512048858 2024-05-09T19:59:36.209291+0000 2024-05-08T20:07:57.553 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6800/792726797 2024-05-09T20:00:11.913247+0000 2024-05-08T20:07:57.553 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/2443488799 2024-05-09T20:00:11.913247+0000 2024-05-08T20:07:57.553 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6801/792726797 2024-05-09T20:00:11.913247+0000 2024-05-08T20:07:57.553 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/2142540104 2024-05-09T20:01:23.760420+0000 2024-05-08T20:07:57.553 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/890224504 2024-05-09T19:59:36.209291+0000 2024-05-08T20:07:57.553 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6801/3694300604 2024-05-09T19:59:36.209291+0000 2024-05-08T20:07:57.554 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/2981330630 2024-05-09T19:59:36.209291+0000 2024-05-08T20:07:57.554 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6800/3694300604 2024-05-09T19:59:36.209291+0000 2024-05-08T20:07:57.554 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6800/3762483871 2024-05-09T20:01:23.760420+0000 2024-05-08T20:07:57.554 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/3109058932 2024-05-09T20:01:23.760420+0000 2024-05-08T20:07:57.554 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/3117466271 2024-05-09T20:00:11.913247+0000 2024-05-08T20:07:57.554 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/1765857891 2024-05-09T20:00:11.913247+0000 2024-05-08T20:07:57.554 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/3612788753 2024-05-09T20:01:23.760420+0000 2024-05-08T20:07:57.554 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6801/3762483871 2024-05-09T20:01:23.760420+0000 2024-05-08T20:07:57.554 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.188:6832/1664076600 2024-05-09T20:07:44.002323+0000 2024-05-08T20:07:57.556 INFO:teuthology.orchestra.run.smithi083.stderr:listed 17 entries 2024-05-08T20:07:57.567 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-05-08T20:07:57.567 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T20:07:57.587 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T20:07:57.954 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:07:57.955 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":70,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","created":"2024-05-08T19:59:07.152446+0000","modified":"2024-05-08T20:07:44.002359+0000","last_up_change":"2024-05-08T20:05:57.759893+0000","last_in_change":"2024-05-08T20:05:40.402230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T20:03:44.088148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.foo.meta","create_time":"2024-05-08T20:07:27.766436+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.data","create_time":"2024-05-08T20:07:28.151095+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.bar.meta","create_time":"2024-05-08T20:07:34.236025+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bar"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs.bar.data","create_time":"2024-05-08T20:07:35.187905+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bar"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78051d3d-3644-4876-af5f-8c956a8491bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6803","nonce":345419579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6805","nonce":345419579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6809","nonce":345419579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6807","nonce":345419579}]},"public_addr":"172.21.15.83:6803/345419579","cluster_addr":"172.21.15.83:6805/345419579","heartbeat_back_addr":"172.21.15.83:6809/345419579","heartbeat_front_addr":"172.21.15.83:6807/345419579","state":["exists","up"]},{"osd":1,"uuid":"bc62b8e9-e30e-41f1-aaee-4c4e356be487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6811","nonce":1992633959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6813","nonce":1992633959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6817","nonce":1992633959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6815","nonce":1992633959}]},"public_addr":"172.21.15.83:6811/1992633959","cluster_addr":"172.21.15.83:6813/1992633959","heartbeat_back_addr":"172.21.15.83:6817/1992633959","heartbeat_front_addr":"172.21.15.83:6815/1992633959","state":["exists","up"]},{"osd":2,"uuid":"4ae0453a-718b-445f-ad00-a3696fec1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6819","nonce":561864588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6821","nonce":561864588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6825","nonce":561864588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6823","nonce":561864588}]},"public_addr":"172.21.15.83:6819/561864588","cluster_addr":"172.21.15.83:6821/561864588","heartbeat_back_addr":"172.21.15.83:6825/561864588","heartbeat_front_addr":"172.21.15.83:6823/561864588","state":["exists","up"]},{"osd":3,"uuid":"2085ec85-187a-4d69-a721-793cfed7d044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6827","nonce":866184273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6829","nonce":866184273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6833","nonce":866184273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6831","nonce":866184273}]},"public_addr":"172.21.15.83:6827/866184273","cluster_addr":"172.21.15.83:6829/866184273","heartbeat_back_addr":"172.21.15.83:6833/866184273","heartbeat_front_addr":"172.21.15.83:6831/866184273","state":["exists","up"]},{"osd":4,"uuid":"b92dad8b-5bd0-4cd1-ba8d-2920927f132e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6801","nonce":422247785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6803","nonce":422247785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6807","nonce":422247785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6805","nonce":422247785}]},"public_addr":"172.21.15.188:6801/422247785","cluster_addr":"172.21.15.188:6803/422247785","heartbeat_back_addr":"172.21.15.188:6807/422247785","heartbeat_front_addr":"172.21.15.188:6805/422247785","state":["exists","up"]},{"osd":5,"uuid":"aeeda050-668f-47a2-90d0-32e1aa35bfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6809","nonce":841652631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6811","nonce":841652631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6815","nonce":841652631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6813","nonce":841652631}]},"public_addr":"172.21.15.188:6809/841652631","cluster_addr":"172.21.15.188:6811/841652631","heartbeat_back_addr":"172.21.15.188:6815/841652631","heartbeat_front_addr":"172.21.15.188:6813/841652631","state":["exists","up"]},{"osd":6,"uuid":"d3720c87-ec2b-490a-937b-a29833f0267c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6817","nonce":118382910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6819","nonce":118382910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6823","nonce":118382910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6821","nonce":118382910}]},"public_addr":"172.21.15.188:6817/118382910","cluster_addr":"172.21.15.188:6819/118382910","heartbeat_back_addr":"172.21.15.188:6823/118382910","heartbeat_front_addr":"172.21.15.188:6821/118382910","state":["exists","up"]},{"osd":7,"uuid":"8fffd2a6-163d-4a9b-adc2-25329a3803ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6825","nonce":3349446159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6827","nonce":3349446159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6831","nonce":3349446159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6829","nonce":3349446159}]},"public_addr":"172.21.15.188:6825/3349446159","cluster_addr":"172.21.15.188:6827/3349446159","heartbeat_back_addr":"172.21.15.188:6831/3349446159","heartbeat_front_addr":"172.21.15.188:6829/3349446159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:02:42.828169+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:11.944380+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:41.150631+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:11.648151+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:36.764713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:03.134028+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:29.964086+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:56.788654+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.188:6833/1664076600":"2024-05-09T20:07:44.002323+0000","172.21.15.83:0/3512048858":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6800/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/2443488799":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6801/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/2142540104":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/890224504":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6801/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/2981330630":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6800/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6800/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3109058932":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3117466271":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/1765857891":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/3612788753":"2024-05-09T20:01:23.760420+0000","172.21.15.83:6801/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.188:6832/1664076600":"2024-05-09T20:07:44.002323+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T20:07:57.969 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T20:03:44.088148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.foo.meta', 'create_time': '2024-05-08T20:07:27.766436+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.foo.data', 'create_time': '2024-05-08T20:07:28.151095+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs.bar.meta', 'create_time': '2024-05-08T20:07:34.236025+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'bar'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs.bar.data', 'create_time': '2024-05-08T20:07:35.187905+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'bar'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T20:07:57.969 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T20:07:58.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:57 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/3093308450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:07:58.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:57 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/3534641063' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-05-08T20:07:58.085 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:57 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/1403386331' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-08T20:07:58.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:57 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3093308450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:07:58.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:57 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3534641063' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-05-08T20:07:58.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:57 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/1403386331' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-08T20:07:58.298 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-05-08T20:07:58.312 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.meta pg_num 2024-05-08T20:07:58.637 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:07:58.651 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.data pg_num 2024-05-08T20:07:58.934 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:58 smithi083 ceph-mon[38695]: pgmap v255: 129 pgs: 129 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-05-08T20:07:58.935 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:58 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2742263797' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:07:58.935 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:58 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/950650465' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-08T20:07:58.935 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:58 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/1644816120' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:07:58.977 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:07:58.991 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.meta pg_num 2024-05-08T20:07:59.318 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:07:59.332 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.data pg_num 2024-05-08T20:07:59.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:58 smithi188 ceph-mon[48761]: pgmap v255: 129 pgs: 129 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-05-08T20:07:59.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:58 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/2742263797' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:07:59.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:58 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/950650465' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-08T20:07:59.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:58 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/1644816120' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:07:59.657 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:07:59.671 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-08T20:07:59.934 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:59 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3361678448' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-05-08T20:07:59.935 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:59 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/722242032' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:07:59.935 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:07:59 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/4036854077' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-05-08T20:08:00.014 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.188:6833/1664076600 2024-05-09T20:07:44.002323+0000 2024-05-08T20:08:00.014 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/3512048858 2024-05-09T19:59:36.209291+0000 2024-05-08T20:08:00.014 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6800/792726797 2024-05-09T20:00:11.913247+0000 2024-05-08T20:08:00.014 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/2443488799 2024-05-09T20:00:11.913247+0000 2024-05-08T20:08:00.014 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6801/792726797 2024-05-09T20:00:11.913247+0000 2024-05-08T20:08:00.014 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/2142540104 2024-05-09T20:01:23.760420+0000 2024-05-08T20:08:00.014 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/890224504 2024-05-09T19:59:36.209291+0000 2024-05-08T20:08:00.014 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6801/3694300604 2024-05-09T19:59:36.209291+0000 2024-05-08T20:08:00.014 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/2981330630 2024-05-09T19:59:36.209291+0000 2024-05-08T20:08:00.015 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6800/3694300604 2024-05-09T19:59:36.209291+0000 2024-05-08T20:08:00.015 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6800/3762483871 2024-05-09T20:01:23.760420+0000 2024-05-08T20:08:00.015 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/3109058932 2024-05-09T20:01:23.760420+0000 2024-05-08T20:08:00.015 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/3117466271 2024-05-09T20:00:11.913247+0000 2024-05-08T20:08:00.015 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/1765857891 2024-05-09T20:00:11.913247+0000 2024-05-08T20:08:00.015 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/3612788753 2024-05-09T20:01:23.760420+0000 2024-05-08T20:08:00.015 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6801/3762483871 2024-05-09T20:01:23.760420+0000 2024-05-08T20:08:00.015 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.188:6832/1664076600 2024-05-09T20:07:44.002323+0000 2024-05-08T20:08:00.016 INFO:teuthology.orchestra.run.smithi083.stderr:listed 17 entries 2024-05-08T20:08:00.028 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi083.front.sepia.ceph.com... 2024-05-08T20:08:00.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:08:00.028 DEBUG:teuthology.orchestra.run.smithi083:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-08T20:08:00.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[107141]: fuse finished with error 0 and tester_r 0 2024-05-08T20:08:00.064 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T20:08:00.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:59 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/3361678448' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-05-08T20:08:00.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:59 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/722242032' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:08:00.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:07:59 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/4036854077' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-05-08T20:08:01.184 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:00 smithi083 ceph-mon[38695]: pgmap v256: 129 pgs: 129 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-05-08T20:08:01.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:00 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/28428855' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-08T20:08:01.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:00 smithi188 ceph-mon[48761]: pgmap v256: 129 pgs: 129 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-05-08T20:08:01.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:00 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/28428855' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-08T20:08:03.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:02 smithi083 ceph-mon[38695]: pgmap v257: 129 pgs: 129 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 170 B/s wr, 0 op/s 2024-05-08T20:08:03.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:02 smithi188 ceph-mon[48761]: pgmap v257: 129 pgs: 129 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 170 B/s wr, 0 op/s 2024-05-08T20:08:05.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:04 smithi083 ceph-mon[38695]: pgmap v258: 129 pgs: 129 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 341 B/s wr, 0 op/s 2024-05-08T20:08:05.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:04 smithi188 ceph-mon[48761]: pgmap v258: 129 pgs: 129 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 341 B/s wr, 0 op/s 2024-05-08T20:08:06.165 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi083.front.sepia.ceph.com 2024-05-08T20:08:06.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:08:06.165 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-08T20:08:06.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:08:06.200 DEBUG:teuthology.orchestra.run.smithi188:> ip netns list 2024-05-08T20:08:06.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:08:06.223 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link delete ceph-brx 2024-05-08T20:08:06.302 INFO:teuthology.orchestra.run.smithi188.stderr:Cannot find device "ceph-brx" 2024-05-08T20:08:06.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T20:08:06.304 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-05-08T20:08:06.304 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d21e4f961318b56f0d915caa2490c3711bf13c34 shell --fsid 526038a6-0d75-11ef-bc97-c7b262605968 -- ceph fs ls 2024-05-08T20:08:06.544 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/mon.smithi083/config 2024-05-08T20:08:06.935 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:06 smithi083 ceph-mon[38695]: pgmap v259: 129 pgs: 129 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-05-08T20:08:07.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:06 smithi188 ceph-mon[48761]: pgmap v259: 129 pgs: 129 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-05-08T20:08:08.868 INFO:teuthology.orchestra.run.smithi083.stdout:name: foo, metadata pool: cephfs.foo.meta, data pools: [cephfs.foo.data ] 2024-05-08T20:08:08.868 INFO:teuthology.orchestra.run.smithi083.stdout:name: bar, metadata pool: cephfs.bar.meta, data pools: [cephfs.bar.data ] 2024-05-08T20:08:09.186 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:08 smithi083 ceph-mon[38695]: pgmap v260: 129 pgs: 129 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-05-08T20:08:09.186 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:08:09.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:08 smithi188 ceph-mon[48761]: pgmap v260: 129 pgs: 129 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-05-08T20:08:09.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:08:09.749 INFO:tasks.cephfs.mount:Mounting Ceph FS bar; just confirmed its presence on cluster 2024-05-08T20:08:09.749 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-08T20:08:09.749 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi188.front.sepia.ceph.com 2024-05-08T20:08:09.749 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-05-08T20:08:09.750 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-05-08T20:08:09.750 INFO:tasks.cephfs.mount:self.cephfs_name = bar 2024-05-08T20:08:09.750 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-08T20:08:09.750 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-08T20:08:09.750 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-05-08T20:08:09.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:08:09.750 DEBUG:teuthology.orchestra.run.smithi188:> ip addr 2024-05-08T20:08:09.772 INFO:teuthology.orchestra.run.smithi188.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-08T20:08:09.772 INFO:teuthology.orchestra.run.smithi188.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-08T20:08:09.773 INFO:teuthology.orchestra.run.smithi188.stdout: inet 127.0.0.1/8 scope host lo 2024-05-08T20:08:09.773 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-05-08T20:08:09.773 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 ::1/128 scope host 2024-05-08T20:08:09.773 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-05-08T20:08:09.773 INFO:teuthology.orchestra.run.smithi188.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T20:08:09.773 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:d9:37:48 brd ff:ff:ff:ff:ff:ff 2024-05-08T20:08:09.773 INFO:teuthology.orchestra.run.smithi188.stdout: altname enp7s0f0 2024-05-08T20:08:09.773 INFO:teuthology.orchestra.run.smithi188.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T20:08:09.773 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:d9:37:49 brd ff:ff:ff:ff:ff:ff 2024-05-08T20:08:09.773 INFO:teuthology.orchestra.run.smithi188.stdout: altname enp7s0f1 2024-05-08T20:08:09.773 INFO:teuthology.orchestra.run.smithi188.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-08T20:08:09.773 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:88:80:38 brd ff:ff:ff:ff:ff:ff 2024-05-08T20:08:09.773 INFO:teuthology.orchestra.run.smithi188.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-08T20:08:09.773 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:88:80:39 brd ff:ff:ff:ff:ff:ff 2024-05-08T20:08:09.774 INFO:teuthology.orchestra.run.smithi188.stdout: inet 172.21.15.188/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-05-08T20:08:09.774 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft 42021sec preferred_lft 42021sec 2024-05-08T20:08:09.774 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::ec4:7aff:fe88:8039/64 scope link 2024-05-08T20:08:09.774 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-05-08T20:08:09.774 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-08T20:08:09.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:08:09.774 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-05-08T20:08:09.774 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-05-08T20:08:09.774 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link add name ceph-brx type bridge 2024-05-08T20:08:09.774 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip addr flush dev ceph-brx 2024-05-08T20:08:09.775 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link set ceph-brx up 2024-05-08T20:08:09.775 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-08T20:08:09.775 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-05-08T20:08:09.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:08:09.966 DEBUG:teuthology.orchestra.run.smithi188:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-08T20:08:09.995 INFO:teuthology.orchestra.run.smithi188.stdout:1 2024-05-08T20:08:09.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:08:09.996 DEBUG:teuthology.orchestra.run.smithi188:> route 2024-05-08T20:08:10.071 INFO:teuthology.orchestra.run.smithi188.stdout:Kernel IP routing table 2024-05-08T20:08:10.071 INFO:teuthology.orchestra.run.smithi188.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-08T20:08:10.071 INFO:teuthology.orchestra.run.smithi188.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-08T20:08:10.071 INFO:teuthology.orchestra.run.smithi188.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-08T20:08:10.071 INFO:teuthology.orchestra.run.smithi188.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-08T20:08:10.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:08:10.072 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-05-08T20:08:10.072 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-05-08T20:08:10.072 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-08T20:08:10.072 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-08T20:08:10.072 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-08T20:08:10.072 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-05-08T20:08:10.151 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:09 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/2347108161' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-08T20:08:10.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:09 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2347108161' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-08T20:08:10.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:08:10.341 DEBUG:teuthology.orchestra.run.smithi188:> ip netns list 2024-05-08T20:08:10.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:08:10.400 DEBUG:teuthology.orchestra.run.smithi188:> ip netns list-id 2024-05-08T20:08:10.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:08:10.457 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-05-08T20:08:10.457 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-05-08T20:08:10.457 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-08T20:08:10.457 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-05-08T20:08:10.457 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-05-08T20:08:10.552 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-05-08T20:08:10.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:08:10.553 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-05-08T20:08:10.553 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-05-08T20:08:10.553 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-05-08T20:08:10.553 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-08T20:08:10.553 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-05-08T20:08:10.553 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-05-08T20:08:10.553 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-05-08T20:08:10.553 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-05-08T20:08:10.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:08:10.740 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-05-08T20:08:10.740 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-05-08T20:08:10.740 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link set brx.0 up 2024-05-08T20:08:10.740 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link set dev brx.0 master ceph-brx 2024-05-08T20:08:10.741 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-05-08T20:08:10.837 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {'cephfs_name': 'bar'} 2024-05-08T20:08:10.837 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-08T20:08:10.838 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-05-08T20:08:10.855 INFO:teuthology.orchestra.run.smithi188.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-05-08T20:08:10.856 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-08T20:08:10.856 DEBUG:teuthology.orchestra.run.smithi188:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-05-08T20:08:10.913 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-05-08T20:08:10.980 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-05-08T20:08:10.980 DEBUG:teuthology.orchestra.run.smithi188:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-05-08T20:08:11.044 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:10 smithi188 ceph-mon[48761]: pgmap v261: 129 pgs: 129 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-05-08T20:08:11.047 INFO:teuthology.orchestra.run.smithi188.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-05-08T20:08:11.048 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-05-08T20:08:11.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:08:11.048 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-05-08T20:08:11.106 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-08T20:08:11.106 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_fs=bar) 2024-05-08T20:08:11.149 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-05-08T20:08:11.175 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-05-08T20:08:11.175 DEBUG:teuthology.orchestra.run.smithi188:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-05-08T20:08:11.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:10 smithi083 ceph-mon[38695]: pgmap v261: 129 pgs: 129 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-05-08T20:08:11.240 INFO:teuthology.orchestra.run.smithi188.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-05-08T20:08:11.241 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-05-08T20:08:11.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:08:11.242 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-05-08T20:08:11.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:2024-05-08T20:08:11.297+0000 7fb3ac2855c0 -1 init, newargv = 0x5561176c56b0 newargc=15 2024-05-08T20:08:11.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[85148]: starting ceph client 2024-05-08T20:08:11.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[85148]: starting fuse 2024-05-08T20:08:11.301 INFO:teuthology.orchestra.run.smithi188.stdout:141 2024-05-08T20:08:11.302 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [141] 2024-05-08T20:08:11.302 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-05-08T20:08:11.302 DEBUG:teuthology.orchestra.run.smithi188:> import glob 2024-05-08T20:08:11.302 DEBUG:teuthology.orchestra.run.smithi188:> import re 2024-05-08T20:08:11.302 DEBUG:teuthology.orchestra.run.smithi188:> import os 2024-05-08T20:08:11.302 DEBUG:teuthology.orchestra.run.smithi188:> import subprocess 2024-05-08T20:08:11.302 DEBUG:teuthology.orchestra.run.smithi188:> 2024-05-08T20:08:11.302 DEBUG:teuthology.orchestra.run.smithi188:> def _find_admin_socket(client_name): 2024-05-08T20:08:11.302 DEBUG:teuthology.orchestra.run.smithi188:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-05-08T20:08:11.303 DEBUG:teuthology.orchestra.run.smithi188:> files = glob.glob(asok_path) 2024-05-08T20:08:11.303 DEBUG:teuthology.orchestra.run.smithi188:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-05-08T20:08:11.303 DEBUG:teuthology.orchestra.run.smithi188:> 2024-05-08T20:08:11.303 DEBUG:teuthology.orchestra.run.smithi188:> # Given a non-glob path, it better be there 2024-05-08T20:08:11.303 DEBUG:teuthology.orchestra.run.smithi188:> if "*" not in asok_path: 2024-05-08T20:08:11.303 DEBUG:teuthology.orchestra.run.smithi188:> assert(len(files) == 1) 2024-05-08T20:08:11.303 DEBUG:teuthology.orchestra.run.smithi188:> return files[0] 2024-05-08T20:08:11.303 DEBUG:teuthology.orchestra.run.smithi188:> 2024-05-08T20:08:11.303 DEBUG:teuthology.orchestra.run.smithi188:> for f in files: 2024-05-08T20:08:11.303 DEBUG:teuthology.orchestra.run.smithi188:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-08T20:08:11.303 DEBUG:teuthology.orchestra.run.smithi188:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-08T20:08:11.303 DEBUG:teuthology.orchestra.run.smithi188:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-08T20:08:11.303 DEBUG:teuthology.orchestra.run.smithi188:> contents = proc_f.read() 2024-05-08T20:08:11.303 DEBUG:teuthology.orchestra.run.smithi188:> if mountpoint in contents: 2024-05-08T20:08:11.303 DEBUG:teuthology.orchestra.run.smithi188:> return f 2024-05-08T20:08:11.303 DEBUG:teuthology.orchestra.run.smithi188:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-08T20:08:11.304 DEBUG:teuthology.orchestra.run.smithi188:> 2024-05-08T20:08:11.304 DEBUG:teuthology.orchestra.run.smithi188:> print(_find_admin_socket("client.1")) 2024-05-08T20:08:11.304 DEBUG:teuthology.orchestra.run.smithi188:> ' 2024-05-08T20:08:11.424 INFO:teuthology.orchestra.run.smithi188.stdout:/var/run/ceph/ceph-client.1.85148.asok 2024-05-08T20:08:11.607 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.85148.asok 2024-05-08T20:08:11.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:08:11.608 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.85148.asok status 2024-05-08T20:08:11.765 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-05-08T20:08:11.765 INFO:teuthology.orchestra.run.smithi188.stdout: "metadata": { 2024-05-08T20:08:11.765 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_sha1": "d21e4f961318b56f0d915caa2490c3711bf13c34", 2024-05-08T20:08:11.765 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_version": "ceph version 18.2.2-1322-gd21e4f96 (d21e4f961318b56f0d915caa2490c3711bf13c34) reef (stable)", 2024-05-08T20:08:11.765 INFO:teuthology.orchestra.run.smithi188.stdout: "entity_id": "1", 2024-05-08T20:08:11.765 INFO:teuthology.orchestra.run.smithi188.stdout: "hostname": "smithi188", 2024-05-08T20:08:11.765 INFO:teuthology.orchestra.run.smithi188.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-05-08T20:08:11.766 INFO:teuthology.orchestra.run.smithi188.stdout: "pid": "85148", 2024-05-08T20:08:11.766 INFO:teuthology.orchestra.run.smithi188.stdout: "root": "/" 2024-05-08T20:08:11.766 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-05-08T20:08:11.766 INFO:teuthology.orchestra.run.smithi188.stdout: "dentry_count": 0, 2024-05-08T20:08:11.766 INFO:teuthology.orchestra.run.smithi188.stdout: "dentry_pinned_count": 0, 2024-05-08T20:08:11.766 INFO:teuthology.orchestra.run.smithi188.stdout: "id": 24415, 2024-05-08T20:08:11.766 INFO:teuthology.orchestra.run.smithi188.stdout: "inst": { 2024-05-08T20:08:11.766 INFO:teuthology.orchestra.run.smithi188.stdout: "name": { 2024-05-08T20:08:11.766 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "client", 2024-05-08T20:08:11.766 INFO:teuthology.orchestra.run.smithi188.stdout: "num": 24415 2024-05-08T20:08:11.766 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-05-08T20:08:11.766 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": { 2024-05-08T20:08:11.766 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "v1", 2024-05-08T20:08:11.767 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": "192.168.0.1:0", 2024-05-08T20:08:11.767 INFO:teuthology.orchestra.run.smithi188.stdout: "nonce": 1070302007 2024-05-08T20:08:11.767 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-05-08T20:08:11.767 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-05-08T20:08:11.767 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": { 2024-05-08T20:08:11.767 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "v1", 2024-05-08T20:08:11.767 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": "192.168.0.1:0", 2024-05-08T20:08:11.767 INFO:teuthology.orchestra.run.smithi188.stdout: "nonce": 1070302007 2024-05-08T20:08:11.767 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-05-08T20:08:11.767 INFO:teuthology.orchestra.run.smithi188.stdout: "inst_str": "client.24415 192.168.0.1:0/1070302007", 2024-05-08T20:08:11.767 INFO:teuthology.orchestra.run.smithi188.stdout: "addr_str": "192.168.0.1:0/1070302007", 2024-05-08T20:08:11.767 INFO:teuthology.orchestra.run.smithi188.stdout: "inode_count": 1, 2024-05-08T20:08:11.768 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_epoch": 14, 2024-05-08T20:08:11.768 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_epoch": 70, 2024-05-08T20:08:11.768 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_epoch_barrier": 0, 2024-05-08T20:08:11.768 INFO:teuthology.orchestra.run.smithi188.stdout: "blocklisted": false, 2024-05-08T20:08:11.768 INFO:teuthology.orchestra.run.smithi188.stdout: "fs_name": "bar" 2024-05-08T20:08:11.768 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-05-08T20:08:11.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:08:11.775 DEBUG:teuthology.orchestra.run.smithi188:> stat --file-system '--printf=%T 2024-05-08T20:08:11.776 DEBUG:teuthology.orchestra.run.smithi188:> ' -- /home/ubuntu/cephtest/mnt.1 2024-05-08T20:08:11.793 INFO:teuthology.orchestra.run.smithi188.stdout:fuseblk 2024-05-08T20:08:11.793 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-05-08T20:08:11.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:08:11.794 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-05-08T20:08:11.861 INFO:teuthology.run_tasks:Running task workunit... 2024-05-08T20:08:11.874 INFO:tasks.workunit:Pulling workunits from ref d21e4f961318b56f0d915caa2490c3711bf13c34 2024-05-08T20:08:11.874 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-08T20:08:11.874 INFO:tasks.workunit:timeout=3h 2024-05-08T20:08:11.874 INFO:tasks.workunit:cleanup=True 2024-05-08T20:08:11.875 DEBUG:teuthology.orchestra.run.smithi083:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-08T20:08:11.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T20:08:11.897 INFO:teuthology.orchestra.run.smithi083.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-08T20:08:11.897 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-05-08T20:08:11.956 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-05-08T20:08:11.957 DEBUG:teuthology.orchestra.run.smithi083:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-05-08T20:08:12.017 DEBUG:teuthology.orchestra.run.smithi188:> stat -- /home/ubuntu/cephtest/mnt.1 2024-05-08T20:08:12.035 INFO:teuthology.orchestra.run.smithi188.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-05-08T20:08:12.036 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-05-08T20:08:12.036 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 8dh/141d Inode: 1 Links: 2 2024-05-08T20:08:12.036 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T20:08:12.036 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:fusefs_t:s0 2024-05-08T20:08:12.036 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-05-08T20:08:12.036 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-05-08 20:07:35.979465266 +0000 2024-05-08T20:08:12.036 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-05-08 20:08:11.869179735 +0000 2024-05-08T20:08:12.036 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2024-05-08T20:08:12.036 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-05-08T20:08:12.036 DEBUG:teuthology.orchestra.run.smithi188:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-05-08T20:08:12.108 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d21e4f961318b56f0d915caa2490c3711bf13c34 2024-05-08T20:08:12.108 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout d21e4f961318b56f0d915caa2490c3711bf13c34 2024-05-08T20:08:12.129 INFO:tasks.workunit.client.0.smithi083.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-08T20:08:12.166 INFO:tasks.workunit.client.1.smithi188.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-05-08T20:08:13.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:12 smithi083 ceph-mon[38695]: pgmap v262: 129 pgs: 129 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-05-08T20:08:13.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:12 smithi188 ceph-mon[48761]: pgmap v262: 129 pgs: 129 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-05-08T20:08:15.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:14 smithi083 ceph-mon[38695]: pgmap v263: 129 pgs: 129 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-05-08T20:08:15.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:14 smithi188 ceph-mon[48761]: pgmap v263: 129 pgs: 129 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-05-08T20:08:17.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:16 smithi083 ceph-mon[38695]: pgmap v264: 129 pgs: 129 active+clean; 586 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-05-08T20:08:17.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:16 smithi188 ceph-mon[48761]: pgmap v264: 129 pgs: 129 active+clean; 586 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-05-08T20:08:19.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:18 smithi083 ceph-mon[38695]: pgmap v265: 129 pgs: 129 active+clean; 586 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-05-08T20:08:19.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:18 smithi188 ceph-mon[48761]: pgmap v265: 129 pgs: 129 active+clean; 586 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-05-08T20:08:21.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:20 smithi083 ceph-mon[38695]: pgmap v266: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 682 B/s wr, 0 op/s 2024-05-08T20:08:21.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:20 smithi188 ceph-mon[48761]: pgmap v266: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 682 B/s wr, 0 op/s 2024-05-08T20:08:23.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:22 smithi083 ceph-mon[38695]: pgmap v267: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-05-08T20:08:23.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:22 smithi188 ceph-mon[48761]: pgmap v267: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-05-08T20:08:24.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]: dispatch 2024-05-08T20:08:24.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-05-08T20:08:24.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-05-08T20:08:24.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-05-08T20:08:24.186 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:08:24.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]: dispatch 2024-05-08T20:08:24.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-05-08T20:08:24.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-05-08T20:08:24.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-05-08T20:08:24.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:08:25.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:24 smithi083 ceph-mon[38695]: pgmap v268: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-05-08T20:08:25.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]': finished 2024-05-08T20:08:25.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-05-08T20:08:25.186 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-05-08T20:08:25.186 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-05-08T20:08:25.186 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:24 smithi083 ceph-mon[38695]: osdmap e71: 8 total, 8 up, 8 in 2024-05-08T20:08:25.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:24 smithi188 ceph-mon[48761]: pgmap v268: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-05-08T20:08:25.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:24 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]': finished 2024-05-08T20:08:25.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:24 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-05-08T20:08:25.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:24 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-05-08T20:08:25.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:24 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-05-08T20:08:25.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:24 smithi188 ceph-mon[48761]: osdmap e71: 8 total, 8 up, 8 in 2024-05-08T20:08:26.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:25 smithi083 ceph-mon[38695]: osdmap e72: 8 total, 8 up, 8 in 2024-05-08T20:08:26.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:25 smithi188 ceph-mon[48761]: osdmap e72: 8 total, 8 up, 8 in 2024-05-08T20:08:27.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:26 smithi083 ceph-mon[38695]: pgmap v271: 129 pgs: 1 activating, 2 peering, 126 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 383 B/s wr, 0 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-08T20:08:27.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:26 smithi188 ceph-mon[48761]: pgmap v271: 129 pgs: 1 activating, 2 peering, 126 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 383 B/s wr, 0 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-08T20:08:29.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:28 smithi083 ceph-mon[38695]: pgmap v272: 129 pgs: 1 activating, 2 peering, 126 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 127 B/s wr, 0 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-08T20:08:29.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:28 smithi188 ceph-mon[48761]: pgmap v272: 129 pgs: 1 activating, 2 peering, 126 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 127 B/s wr, 0 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-08T20:08:31.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:30 smithi083 ceph-mon[38695]: pgmap v273: 129 pgs: 1 activating, 128 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-08T20:08:31.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:30 smithi188 ceph-mon[48761]: pgmap v273: 129 pgs: 1 activating, 128 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-08T20:08:33.181 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:32 smithi188 ceph-mon[48761]: pgmap v274: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-08T20:08:33.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:32 smithi083 ceph-mon[38695]: pgmap v274: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-08T20:08:35.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:34 smithi083 ceph-mon[38695]: pgmap v275: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-08T20:08:35.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:34 smithi188 ceph-mon[48761]: pgmap v275: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-08T20:08:37.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:36 smithi083 ceph-mon[38695]: pgmap v276: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-08T20:08:37.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:36 smithi188 ceph-mon[48761]: pgmap v276: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-08T20:08:39.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:38 smithi083 ceph-mon[38695]: pgmap v277: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-05-08T20:08:39.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:08:39.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:38 smithi188 ceph-mon[48761]: pgmap v277: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-05-08T20:08:39.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:08:41.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:40 smithi083 ceph-mon[38695]: pgmap v278: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-05-08T20:08:41.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:40 smithi188 ceph-mon[48761]: pgmap v278: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-05-08T20:08:43.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:42 smithi083 ceph-mon[38695]: pgmap v279: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:08:43.294 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:42 smithi188 ceph-mon[48761]: pgmap v279: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:08:45.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:44 smithi083 ceph-mon[38695]: pgmap v280: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:08:45.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:44 smithi188 ceph-mon[48761]: pgmap v280: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:08:47.185 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:46 smithi083 ceph-mon[38695]: pgmap v281: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:08:47.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:46 smithi188 ceph-mon[48761]: pgmap v281: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:08:49.186 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:48 smithi083 ceph-mon[38695]: pgmap v282: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:08:49.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:48 smithi188 ceph-mon[48761]: pgmap v282: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:08:51.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:50 smithi188 ceph-mon[48761]: pgmap v283: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:08:51.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:50 smithi083 ceph-mon[38695]: pgmap v283: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:08:53.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:52 smithi188 ceph-mon[48761]: pgmap v284: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:08:53.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:52 smithi083 ceph-mon[38695]: pgmap v284: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:08:54.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:53 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:08:54.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:53 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:08:55.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:54 smithi188 ceph-mon[48761]: pgmap v285: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:08:55.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:08:55.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:08:55.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:08:55.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:08:55.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:54 smithi083 ceph-mon[38695]: pgmap v285: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:08:55.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:08:55.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:08:55.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:08:55.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:08:57.321 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:56 smithi083 ceph-mon[38695]: pgmap v286: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:08:57.335 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:56 smithi188 ceph-mon[48761]: pgmap v286: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:08:59.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:08:58 smithi188 ceph-mon[48761]: pgmap v287: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:08:59.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:08:58 smithi083 ceph-mon[38695]: pgmap v287: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:01.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:00 smithi188 ceph-mon[48761]: pgmap v288: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:01.435 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:00 smithi083 ceph-mon[38695]: pgmap v288: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:03.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:02 smithi188 ceph-mon[48761]: pgmap v289: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:03.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:02 smithi083 ceph-mon[38695]: pgmap v289: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:05.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:04 smithi188 ceph-mon[48761]: pgmap v290: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:05.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:04 smithi083 ceph-mon[38695]: pgmap v290: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:07.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:06 smithi188 ceph-mon[48761]: pgmap v291: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:07.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:06 smithi083 ceph-mon[38695]: pgmap v291: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:09.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:08 smithi188 ceph-mon[48761]: pgmap v292: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:09.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:09:09.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:08 smithi083 ceph-mon[38695]: pgmap v292: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:09.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:09:11.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:10 smithi188 ceph-mon[48761]: pgmap v293: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:11.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:10 smithi083 ceph-mon[38695]: pgmap v293: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:13.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:12 smithi188 ceph-mon[48761]: pgmap v294: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:13.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:12 smithi083 ceph-mon[38695]: pgmap v294: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:15.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:14 smithi188 ceph-mon[48761]: pgmap v295: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:15.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:14 smithi083 ceph-mon[38695]: pgmap v295: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:17.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:16 smithi188 ceph-mon[48761]: pgmap v296: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:17.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:16 smithi083 ceph-mon[38695]: pgmap v296: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:19.086 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:18 smithi188 ceph-mon[48761]: pgmap v297: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:19.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:18 smithi083 ceph-mon[38695]: pgmap v297: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:21.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:20 smithi188 ceph-mon[48761]: pgmap v298: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:21.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:20 smithi083 ceph-mon[38695]: pgmap v298: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:22.977 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:22 smithi083 ceph-mon[38695]: pgmap v299: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:23.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:22 smithi188 ceph-mon[48761]: pgmap v299: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:24.086 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:09:24.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:09:25.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:24 smithi188 ceph-mon[48761]: pgmap v300: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:25.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:24 smithi083 ceph-mon[38695]: pgmap v300: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:27.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:26 smithi188 ceph-mon[48761]: pgmap v301: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:27.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:26 smithi083 ceph-mon[38695]: pgmap v301: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:29.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:28 smithi188 ceph-mon[48761]: pgmap v302: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:29.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:28 smithi083 ceph-mon[38695]: pgmap v302: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:31.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:30 smithi188 ceph-mon[48761]: pgmap v303: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:31.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:30 smithi083 ceph-mon[38695]: pgmap v303: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:33.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:33 smithi188 ceph-mon[48761]: pgmap v304: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:33.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:33 smithi083 ceph-mon[38695]: pgmap v304: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:35.210 INFO:tasks.workunit.client.1.smithi188.stderr:Note: switching to 'd21e4f961318b56f0d915caa2490c3711bf13c34'. 2024-05-08T20:09:35.210 INFO:tasks.workunit.client.1.smithi188.stderr: 2024-05-08T20:09:35.210 INFO:tasks.workunit.client.1.smithi188.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-08T20:09:35.210 INFO:tasks.workunit.client.1.smithi188.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-08T20:09:35.210 INFO:tasks.workunit.client.1.smithi188.stderr:state without impacting any branches by switching back to a branch. 2024-05-08T20:09:35.210 INFO:tasks.workunit.client.1.smithi188.stderr: 2024-05-08T20:09:35.210 INFO:tasks.workunit.client.1.smithi188.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-08T20:09:35.210 INFO:tasks.workunit.client.1.smithi188.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-08T20:09:35.211 INFO:tasks.workunit.client.1.smithi188.stderr: 2024-05-08T20:09:35.211 INFO:tasks.workunit.client.1.smithi188.stderr: git switch -c 2024-05-08T20:09:35.211 INFO:tasks.workunit.client.1.smithi188.stderr: 2024-05-08T20:09:35.211 INFO:tasks.workunit.client.1.smithi188.stderr:Or undo this operation with: 2024-05-08T20:09:35.211 INFO:tasks.workunit.client.1.smithi188.stderr: 2024-05-08T20:09:35.211 INFO:tasks.workunit.client.1.smithi188.stderr: git switch - 2024-05-08T20:09:35.211 INFO:tasks.workunit.client.1.smithi188.stderr: 2024-05-08T20:09:35.211 INFO:tasks.workunit.client.1.smithi188.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-08T20:09:35.211 INFO:tasks.workunit.client.1.smithi188.stderr: 2024-05-08T20:09:35.211 INFO:tasks.workunit.client.1.smithi188.stderr:HEAD is now at d21e4f96131 Merge PR #57343 into wip-pdonnell-testing-20240508.150423-reef 2024-05-08T20:09:35.216 DEBUG:teuthology.orchestra.run.smithi188:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-05-08T20:09:35.288 INFO:tasks.workunit.client.1.smithi188.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-08T20:09:35.290 INFO:tasks.workunit.client.1.smithi188.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-05-08T20:09:35.290 INFO:tasks.workunit.client.1.smithi188.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-08T20:09:35.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:35 smithi188 ceph-mon[48761]: pgmap v305: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:35.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:35 smithi083 ceph-mon[38695]: pgmap v305: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:35.492 INFO:tasks.workunit.client.0.smithi083.stderr:Note: switching to 'd21e4f961318b56f0d915caa2490c3711bf13c34'. 2024-05-08T20:09:35.492 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-05-08T20:09:35.492 INFO:tasks.workunit.client.0.smithi083.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-08T20:09:35.492 INFO:tasks.workunit.client.0.smithi083.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-08T20:09:35.492 INFO:tasks.workunit.client.0.smithi083.stderr:state without impacting any branches by switching back to a branch. 2024-05-08T20:09:35.492 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-05-08T20:09:35.492 INFO:tasks.workunit.client.0.smithi083.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-08T20:09:35.492 INFO:tasks.workunit.client.0.smithi083.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-08T20:09:35.492 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-05-08T20:09:35.493 INFO:tasks.workunit.client.0.smithi083.stderr: git switch -c 2024-05-08T20:09:35.493 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-05-08T20:09:35.493 INFO:tasks.workunit.client.0.smithi083.stderr:Or undo this operation with: 2024-05-08T20:09:35.493 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-05-08T20:09:35.493 INFO:tasks.workunit.client.0.smithi083.stderr: git switch - 2024-05-08T20:09:35.493 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-05-08T20:09:35.493 INFO:tasks.workunit.client.0.smithi083.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-08T20:09:35.493 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-05-08T20:09:35.493 INFO:tasks.workunit.client.0.smithi083.stderr:HEAD is now at d21e4f96131 Merge PR #57343 into wip-pdonnell-testing-20240508.150423-reef 2024-05-08T20:09:35.498 DEBUG:teuthology.orchestra.run.smithi083:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-08T20:09:35.561 INFO:tasks.workunit.client.0.smithi083.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-08T20:09:35.563 INFO:tasks.workunit.client.0.smithi083.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-08T20:09:35.563 INFO:tasks.workunit.client.0.smithi083.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-08T20:09:35.835 INFO:tasks.workunit.client.1.smithi188.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-08T20:09:35.900 INFO:tasks.workunit.client.1.smithi188.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-08T20:09:35.938 INFO:tasks.workunit.client.1.smithi188.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-05-08T20:09:35.940 INFO:tasks.workunit.client.1.smithi188.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-05-08T20:09:35.940 INFO:tasks.workunit.client.1.smithi188.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-08T20:09:35.978 INFO:tasks.workunit.client.1.smithi188.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-05-08T20:09:35.981 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-05-08T20:09:35.981 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-05-08T20:09:36.039 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.1... 2024-05-08T20:09:36.040 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-05-08T20:09:36.040 DEBUG:teuthology.orchestra.run.smithi188:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d21e4f961318b56f0d915caa2490c3711bf13c34 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/dbench.sh 2024-05-08T20:09:36.109 INFO:tasks.workunit.client.1.smithi188.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-05-08T20:09:36.109 INFO:tasks.workunit.client.1.smithi188.stdout: 2024-05-08T20:09:36.109 INFO:tasks.workunit.client.1.smithi188.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-05-08T20:09:36.117 INFO:tasks.workunit.client.1.smithi188.stdout:0 of 1 processes prepared for launch 0 sec 2024-05-08T20:09:36.160 INFO:tasks.workunit.client.0.smithi083.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-08T20:09:36.213 INFO:tasks.workunit.client.0.smithi083.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-08T20:09:36.217 INFO:tasks.workunit.client.1.smithi188.stdout:1 of 1 processes prepared for launch 0 sec 2024-05-08T20:09:36.217 INFO:tasks.workunit.client.1.smithi188.stdout:releasing clients 2024-05-08T20:09:36.251 INFO:tasks.workunit.client.0.smithi083.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-08T20:09:36.252 INFO:tasks.workunit.client.0.smithi083.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-08T20:09:36.252 INFO:tasks.workunit.client.0.smithi083.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-08T20:09:36.295 INFO:tasks.workunit.client.0.smithi083.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-08T20:09:36.299 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-05-08T20:09:36.299 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-08T20:09:36.361 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-05-08T20:09:36.363 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-05-08T20:09:36.364 DEBUG:teuthology.orchestra.run.smithi083:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d21e4f961318b56f0d915caa2490c3711bf13c34 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-05-08T20:09:36.438 INFO:tasks.workunit.client.0.smithi083.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-05-08T20:09:36.438 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-05-08T20:09:36.438 INFO:tasks.workunit.client.0.smithi083.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-05-08T20:09:36.449 INFO:tasks.workunit.client.0.smithi083.stdout:0 of 1 processes prepared for launch 0 sec 2024-05-08T20:09:36.549 INFO:tasks.workunit.client.0.smithi083.stdout:1 of 1 processes prepared for launch 0 sec 2024-05-08T20:09:36.549 INFO:tasks.workunit.client.0.smithi083.stdout:releasing clients 2024-05-08T20:09:37.218 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1967 26.45 MB/sec warmup 1 sec latency 4.102 ms 2024-05-08T20:09:37.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:37 smithi188 ceph-mon[48761]: pgmap v306: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:37.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:37 smithi083 ceph-mon[38695]: pgmap v306: 129 pgs: 129 active+clean; 587 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-08T20:09:37.549 INFO:tasks.workunit.client.0.smithi083.stdout: 1 3000 33.54 MB/sec warmup 1 sec latency 106.667 ms 2024-05-08T20:09:38.218 INFO:tasks.workunit.client.1.smithi188.stdout: 1 3838 19.88 MB/sec warmup 2 sec latency 4.560 ms 2024-05-08T20:09:38.549 INFO:tasks.workunit.client.0.smithi083.stdout: 1 6477 26.30 MB/sec warmup 2 sec latency 61.922 ms 2024-05-08T20:09:39.218 INFO:tasks.workunit.client.1.smithi188.stdout: 1 5438 15.16 MB/sec warmup 3 sec latency 5.028 ms 2024-05-08T20:09:39.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:39 smithi188 ceph-mon[48761]: pgmap v307: 129 pgs: 129 active+clean; 3.7 MiB data, 237 MiB used, 715 GiB / 715 GiB avail; 287 KiB/s wr, 1 op/s 2024-05-08T20:09:39.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:09:39.438 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:39 smithi083 ceph-mon[38695]: pgmap v307: 129 pgs: 129 active+clean; 3.7 MiB data, 237 MiB used, 715 GiB / 715 GiB avail; 287 KiB/s wr, 1 op/s 2024-05-08T20:09:39.438 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:09:39.549 INFO:tasks.workunit.client.0.smithi083.stdout: 1 9581 22.32 MB/sec warmup 3 sec latency 58.366 ms 2024-05-08T20:09:40.218 INFO:tasks.workunit.client.1.smithi188.stdout: 1 7356 14.72 MB/sec warmup 4 sec latency 3.861 ms 2024-05-08T20:09:40.549 INFO:tasks.workunit.client.0.smithi083.stdout: 1 12655 20.97 MB/sec warmup 4 sec latency 58.368 ms 2024-05-08T20:09:41.218 INFO:tasks.workunit.client.1.smithi188.stdout: 1 8948 12.92 MB/sec warmup 5 sec latency 4.039 ms 2024-05-08T20:09:41.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:41 smithi188 ceph-mon[48761]: pgmap v308: 129 pgs: 129 active+clean; 22 MiB data, 293 MiB used, 715 GiB / 715 GiB avail; 2.1 MiB/s wr, 46 op/s 2024-05-08T20:09:41.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:41 smithi083 ceph-mon[38695]: pgmap v308: 129 pgs: 129 active+clean; 22 MiB data, 293 MiB used, 715 GiB / 715 GiB avail; 2.1 MiB/s wr, 46 op/s 2024-05-08T20:09:41.549 INFO:tasks.workunit.client.0.smithi083.stdout: 1 16087 20.57 MB/sec warmup 5 sec latency 52.408 ms 2024-05-08T20:09:42.218 INFO:tasks.workunit.client.1.smithi188.stdout: 1 10869 13.00 MB/sec warmup 6 sec latency 8.526 ms 2024-05-08T20:09:42.549 INFO:tasks.workunit.client.0.smithi083.stdout: 1 19384 20.30 MB/sec warmup 6 sec latency 60.878 ms 2024-05-08T20:09:43.218 INFO:tasks.workunit.client.1.smithi188.stdout: 1 12391 11.95 MB/sec warmup 7 sec latency 13.565 ms 2024-05-08T20:09:43.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:43 smithi188 ceph-mon[48761]: pgmap v309: 129 pgs: 129 active+clean; 27 MiB data, 313 MiB used, 715 GiB / 715 GiB avail; 2.6 MiB/s wr, 53 op/s 2024-05-08T20:09:43.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:43 smithi083 ceph-mon[38695]: pgmap v309: 129 pgs: 129 active+clean; 27 MiB data, 313 MiB used, 715 GiB / 715 GiB avail; 2.6 MiB/s wr, 53 op/s 2024-05-08T20:09:43.549 INFO:tasks.workunit.client.0.smithi083.stdout: 1 22962 20.13 MB/sec warmup 7 sec latency 58.308 ms 2024-05-08T20:09:44.218 INFO:tasks.workunit.client.1.smithi188.stdout: 1 14304 12.01 MB/sec warmup 8 sec latency 4.006 ms 2024-05-08T20:09:44.549 INFO:tasks.workunit.client.0.smithi083.stdout: 1 26363 19.99 MB/sec warmup 8 sec latency 56.122 ms 2024-05-08T20:09:45.218 INFO:tasks.workunit.client.1.smithi188.stdout: 1 15842 11.41 MB/sec warmup 9 sec latency 13.692 ms 2024-05-08T20:09:45.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:45 smithi188 ceph-mon[48761]: pgmap v310: 129 pgs: 129 active+clean; 29 MiB data, 321 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 3.1 MiB/s wr, 69 op/s 2024-05-08T20:09:45.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:45 smithi083 ceph-mon[38695]: pgmap v310: 129 pgs: 129 active+clean; 29 MiB data, 321 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 3.1 MiB/s wr, 69 op/s 2024-05-08T20:09:45.550 INFO:tasks.workunit.client.0.smithi083.stdout: 1 29802 19.89 MB/sec warmup 9 sec latency 56.071 ms 2024-05-08T20:09:46.218 INFO:tasks.workunit.client.1.smithi188.stdout: 1 17751 11.49 MB/sec warmup 10 sec latency 8.703 ms 2024-05-08T20:09:46.550 INFO:tasks.workunit.client.0.smithi083.stdout: 1 32954 19.40 MB/sec warmup 10 sec latency 58.473 ms 2024-05-08T20:09:47.218 INFO:tasks.workunit.client.1.smithi188.stdout: 1 19287 11.06 MB/sec warmup 11 sec latency 6.774 ms 2024-05-08T20:09:47.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:47 smithi188 ceph-mon[48761]: pgmap v311: 129 pgs: 129 active+clean; 31 MiB data, 334 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 4.3 MiB/s wr, 123 op/s 2024-05-08T20:09:47.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:47 smithi083 ceph-mon[38695]: pgmap v311: 129 pgs: 129 active+clean; 31 MiB data, 334 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 4.3 MiB/s wr, 123 op/s 2024-05-08T20:09:47.550 INFO:tasks.workunit.client.0.smithi083.stdout: 1 36394 19.36 MB/sec warmup 11 sec latency 50.159 ms 2024-05-08T20:09:48.218 INFO:tasks.workunit.client.1.smithi188.stdout: 1 21169 10.93 MB/sec warmup 12 sec latency 9.469 ms 2024-05-08T20:09:48.550 INFO:tasks.workunit.client.0.smithi083.stdout: 1 39942 19.34 MB/sec warmup 12 sec latency 52.564 ms 2024-05-08T20:09:49.218 INFO:tasks.workunit.client.1.smithi188.stdout: 1 22759 10.83 MB/sec warmup 13 sec latency 8.944 ms 2024-05-08T20:09:49.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:49 smithi188 ceph-mon[48761]: pgmap v312: 129 pgs: 129 active+clean; 31 MiB data, 335 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 4.5 MiB/s wr, 129 op/s 2024-05-08T20:09:49.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:49 smithi083 ceph-mon[38695]: pgmap v312: 129 pgs: 129 active+clean; 31 MiB data, 335 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 4.5 MiB/s wr, 129 op/s 2024-05-08T20:09:49.550 INFO:tasks.workunit.client.0.smithi083.stdout: 1 43398 19.30 MB/sec warmup 13 sec latency 88.115 ms 2024-05-08T20:09:50.218 INFO:tasks.workunit.client.1.smithi188.stdout: 1 24652 10.73 MB/sec warmup 14 sec latency 5.550 ms 2024-05-08T20:09:50.550 INFO:tasks.workunit.client.0.smithi083.stdout: 1 46892 19.28 MB/sec warmup 14 sec latency 61.259 ms 2024-05-08T20:09:51.218 INFO:tasks.workunit.client.1.smithi188.stdout: 1 26258 10.66 MB/sec warmup 15 sec latency 7.987 ms 2024-05-08T20:09:51.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:51 smithi188 ceph-mon[48761]: pgmap v313: 129 pgs: 129 active+clean; 33 MiB data, 354 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5.4 MiB/s wr, 196 op/s 2024-05-08T20:09:51.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:51 smithi083 ceph-mon[38695]: pgmap v313: 129 pgs: 129 active+clean; 33 MiB data, 354 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5.4 MiB/s wr, 196 op/s 2024-05-08T20:09:51.550 INFO:tasks.workunit.client.0.smithi083.stdout: 1 50423 19.27 MB/sec warmup 15 sec latency 56.821 ms 2024-05-08T20:09:52.218 INFO:tasks.workunit.client.1.smithi188.stdout: 1 28182 10.58 MB/sec warmup 16 sec latency 4.456 ms 2024-05-08T20:09:52.550 INFO:tasks.workunit.client.0.smithi083.stdout: 1 53915 19.25 MB/sec warmup 16 sec latency 48.864 ms 2024-05-08T20:09:53.218 INFO:tasks.workunit.client.1.smithi188.stdout: 1 29638 10.36 MB/sec warmup 17 sec latency 8.233 ms 2024-05-08T20:09:53.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:53 smithi188 ceph-mon[48761]: pgmap v314: 129 pgs: 129 active+clean; 33 MiB data, 357 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 165 op/s 2024-05-08T20:09:53.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:53 smithi083 ceph-mon[38695]: pgmap v314: 129 pgs: 129 active+clean; 33 MiB data, 357 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 165 op/s 2024-05-08T20:09:53.550 INFO:tasks.workunit.client.0.smithi083.stdout: 1 56432 18.89 MB/sec warmup 17 sec latency 60.134 ms 2024-05-08T20:09:54.218 INFO:tasks.workunit.client.1.smithi188.stdout: 1 31705 10.46 MB/sec warmup 18 sec latency 7.189 ms 2024-05-08T20:09:54.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:09:54.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:09:54.550 INFO:tasks.workunit.client.0.smithi083.stdout: 1 59157 18.55 MB/sec warmup 18 sec latency 60.824 ms 2024-05-08T20:09:55.218 INFO:tasks.workunit.client.1.smithi188.stdout: 1 33057 10.27 MB/sec warmup 19 sec latency 12.989 ms 2024-05-08T20:09:55.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:55 smithi188 ceph-mon[48761]: pgmap v315: 129 pgs: 129 active+clean; 35 MiB data, 366 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.1 MiB/s wr, 184 op/s 2024-05-08T20:09:55.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:55 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:09:55.436 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:55 smithi083 ceph-mon[38695]: pgmap v315: 129 pgs: 129 active+clean; 35 MiB data, 366 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.1 MiB/s wr, 184 op/s 2024-05-08T20:09:55.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:55 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:09:55.550 INFO:tasks.workunit.client.0.smithi083.stdout: 1 61967 18.49 MB/sec warmup 19 sec latency 51.142 ms 2024-05-08T20:09:56.218 INFO:tasks.workunit.client.1.smithi188.stdout: 1 34965 10.28 MB/sec warmup 20 sec latency 8.916 ms 2024-05-08T20:09:56.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:56 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:09:56.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:56 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:09:56.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:56 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:09:56.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:56 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:09:56.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:56 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:09:56.550 INFO:tasks.workunit.client.0.smithi083.stdout: 1 64793 18.30 MB/sec warmup 20 sec latency 57.940 ms 2024-05-08T20:09:56.587 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:56 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:09:56.587 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:56 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:09:56.587 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:56 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:09:56.587 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:56 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:09:56.587 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:56 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:09:57.218 INFO:tasks.workunit.client.1.smithi188.stdout: 1 36483 10.15 MB/sec warmup 21 sec latency 7.492 ms 2024-05-08T20:09:57.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:57 smithi083 ceph-mon[38695]: pgmap v316: 129 pgs: 129 active+clean; 38 MiB data, 381 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.8 MiB/s wr, 237 op/s 2024-05-08T20:09:57.550 INFO:tasks.workunit.client.0.smithi083.stdout: 1 68223 18.32 MB/sec warmup 21 sec latency 60.437 ms 2024-05-08T20:09:57.587 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:57 smithi188 ceph-mon[48761]: pgmap v316: 129 pgs: 129 active+clean; 38 MiB data, 381 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.8 MiB/s wr, 237 op/s 2024-05-08T20:09:58.218 INFO:tasks.workunit.client.1.smithi188.stdout: 1 38404 10.19 MB/sec warmup 22 sec latency 7.747 ms 2024-05-08T20:09:58.550 INFO:tasks.workunit.client.0.smithi083.stdout: 1 71648 18.35 MB/sec warmup 22 sec latency 58.136 ms 2024-05-08T20:09:59.219 INFO:tasks.workunit.client.1.smithi188.stdout: 1 39920 10.08 MB/sec warmup 23 sec latency 12.287 ms 2024-05-08T20:09:59.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:09:59 smithi083 ceph-mon[38695]: pgmap v317: 129 pgs: 129 active+clean; 38 MiB data, 385 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 195 op/s 2024-05-08T20:09:59.550 INFO:tasks.workunit.client.0.smithi083.stdout: 1 74298 18.17 MB/sec warmup 23 sec latency 66.775 ms 2024-05-08T20:09:59.587 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:09:59 smithi188 ceph-mon[48761]: pgmap v317: 129 pgs: 129 active+clean; 38 MiB data, 385 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 195 op/s 2024-05-08T20:10:00.219 INFO:tasks.workunit.client.1.smithi188.stdout: 1 41665 9.97 MB/sec warmup 24 sec latency 5.144 ms 2024-05-08T20:10:00.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:00 smithi083 ceph-mon[38695]: overall HEALTH_OK 2024-05-08T20:10:00.550 INFO:tasks.workunit.client.0.smithi083.stdout: 1 76435 17.83 MB/sec warmup 24 sec latency 152.793 ms 2024-05-08T20:10:00.587 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:00 smithi188 ceph-mon[48761]: overall HEALTH_OK 2024-05-08T20:10:01.219 INFO:tasks.workunit.client.1.smithi188.stdout: 1 43337 10.03 MB/sec warmup 25 sec latency 5.590 ms 2024-05-08T20:10:01.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:01 smithi083 ceph-mon[38695]: pgmap v318: 129 pgs: 129 active+clean; 37 MiB data, 395 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.6 MiB/s wr, 218 op/s 2024-05-08T20:10:01.550 INFO:tasks.workunit.client.0.smithi083.stdout: 1 76890 17.18 MB/sec warmup 25 sec latency 575.241 ms 2024-05-08T20:10:01.587 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:01 smithi188 ceph-mon[48761]: pgmap v318: 129 pgs: 129 active+clean; 37 MiB data, 395 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.6 MiB/s wr, 218 op/s 2024-05-08T20:10:02.219 INFO:tasks.workunit.client.1.smithi188.stdout: 1 45097 9.93 MB/sec warmup 26 sec latency 4.910 ms 2024-05-08T20:10:02.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:02 smithi083 ceph-mon[38695]: pgmap v319: 129 pgs: 129 active+clean; 37 MiB data, 397 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.8 MiB/s wr, 170 op/s 2024-05-08T20:10:02.550 INFO:tasks.workunit.client.0.smithi083.stdout: 1 79724 17.19 MB/sec warmup 26 sec latency 356.516 ms 2024-05-08T20:10:02.587 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:02 smithi188 ceph-mon[48761]: pgmap v319: 129 pgs: 129 active+clean; 37 MiB data, 397 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.8 MiB/s wr, 170 op/s 2024-05-08T20:10:03.219 INFO:tasks.workunit.client.1.smithi188.stdout: 1 46804 9.99 MB/sec warmup 27 sec latency 10.488 ms 2024-05-08T20:10:03.550 INFO:tasks.workunit.client.0.smithi083.stdout: 1 82625 17.10 MB/sec warmup 27 sec latency 51.126 ms 2024-05-08T20:10:04.219 INFO:tasks.workunit.client.1.smithi188.stdout: 1 48588 9.88 MB/sec warmup 28 sec latency 4.028 ms 2024-05-08T20:10:04.551 INFO:tasks.workunit.client.0.smithi083.stdout: 1 86664 17.32 MB/sec warmup 28 sec latency 57.573 ms 2024-05-08T20:10:05.187 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:04 smithi083 ceph-mon[38695]: pgmap v320: 129 pgs: 129 active+clean; 41 MiB data, 405 MiB used, 715 GiB / 715 GiB avail; 9.6 KiB/s rd, 4.2 MiB/s wr, 215 op/s 2024-05-08T20:10:05.219 INFO:tasks.workunit.client.1.smithi188.stdout: 1 50248 9.96 MB/sec warmup 29 sec latency 6.883 ms 2024-05-08T20:10:05.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:04 smithi188 ceph-mon[48761]: pgmap v320: 129 pgs: 129 active+clean; 41 MiB data, 405 MiB used, 715 GiB / 715 GiB avail; 9.6 KiB/s rd, 4.2 MiB/s wr, 215 op/s 2024-05-08T20:10:05.551 INFO:tasks.workunit.client.0.smithi083.stdout: 1 89652 17.24 MB/sec warmup 29 sec latency 50.445 ms 2024-05-08T20:10:06.219 INFO:tasks.workunit.client.1.smithi188.stdout: 1 52054 9.85 MB/sec warmup 30 sec latency 3.926 ms 2024-05-08T20:10:06.551 INFO:tasks.workunit.client.0.smithi083.stdout: 1 93082 17.29 MB/sec warmup 30 sec latency 57.466 ms 2024-05-08T20:10:07.187 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:06 smithi083 ceph-mon[38695]: pgmap v321: 129 pgs: 129 active+clean; 41 MiB data, 407 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 216 op/s 2024-05-08T20:10:07.219 INFO:tasks.workunit.client.1.smithi188.stdout: 1 53720 9.93 MB/sec warmup 31 sec latency 7.820 ms 2024-05-08T20:10:07.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:06 smithi188 ceph-mon[48761]: pgmap v321: 129 pgs: 129 active+clean; 41 MiB data, 407 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 216 op/s 2024-05-08T20:10:07.551 INFO:tasks.workunit.client.0.smithi083.stdout: 1 95564 17.18 MB/sec warmup 31 sec latency 284.505 ms 2024-05-08T20:10:08.219 INFO:tasks.workunit.client.1.smithi188.stdout: 1 55464 9.80 MB/sec warmup 32 sec latency 6.543 ms 2024-05-08T20:10:08.551 INFO:tasks.workunit.client.0.smithi083.stdout: 1 96498 16.80 MB/sec warmup 32 sec latency 238.180 ms 2024-05-08T20:10:09.187 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:08 smithi083 ceph-mon[38695]: pgmap v322: 129 pgs: 129 active+clean; 41 MiB data, 410 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.5 MiB/s wr, 159 op/s 2024-05-08T20:10:09.187 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:10:09.219 INFO:tasks.workunit.client.1.smithi188.stdout: 1 57187 9.91 MB/sec warmup 33 sec latency 5.754 ms 2024-05-08T20:10:09.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:08 smithi188 ceph-mon[48761]: pgmap v322: 129 pgs: 129 active+clean; 41 MiB data, 410 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.5 MiB/s wr, 159 op/s 2024-05-08T20:10:09.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:10:09.551 INFO:tasks.workunit.client.0.smithi083.stdout: 1 96558 16.29 MB/sec warmup 33 sec latency 218.812 ms 2024-05-08T20:10:10.219 INFO:tasks.workunit.client.1.smithi188.stdout: 1 58871 9.78 MB/sec warmup 34 sec latency 9.390 ms 2024-05-08T20:10:10.551 INFO:tasks.workunit.client.0.smithi083.stdout: 1 96626 15.82 MB/sec warmup 34 sec latency 224.199 ms 2024-05-08T20:10:11.187 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:10 smithi083 ceph-mon[38695]: pgmap v323: 129 pgs: 129 active+clean; 44 MiB data, 443 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.4 MiB/s wr, 214 op/s 2024-05-08T20:10:11.219 INFO:tasks.workunit.client.1.smithi188.stdout: 1 60690 9.89 MB/sec warmup 35 sec latency 5.897 ms 2024-05-08T20:10:11.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:10 smithi188 ceph-mon[48761]: pgmap v323: 129 pgs: 129 active+clean; 44 MiB data, 443 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.4 MiB/s wr, 214 op/s 2024-05-08T20:10:11.551 INFO:tasks.workunit.client.0.smithi083.stdout: 1 96743 15.38 MB/sec warmup 35 sec latency 298.495 ms 2024-05-08T20:10:12.219 INFO:tasks.workunit.client.1.smithi188.stdout: 1 62368 9.77 MB/sec warmup 36 sec latency 8.854 ms 2024-05-08T20:10:12.551 INFO:tasks.workunit.client.0.smithi083.stdout: 1 100106 15.47 MB/sec warmup 36 sec latency 211.013 ms 2024-05-08T20:10:13.187 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:12 smithi083 ceph-mon[38695]: pgmap v324: 129 pgs: 129 active+clean; 45 MiB data, 451 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 205 op/s 2024-05-08T20:10:13.219 INFO:tasks.workunit.client.1.smithi188.stdout: 1 64185 9.86 MB/sec warmup 37 sec latency 3.986 ms 2024-05-08T20:10:13.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:12 smithi188 ceph-mon[48761]: pgmap v324: 129 pgs: 129 active+clean; 45 MiB data, 451 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 205 op/s 2024-05-08T20:10:13.551 INFO:tasks.workunit.client.0.smithi083.stdout: 1 103642 15.57 MB/sec warmup 37 sec latency 59.793 ms 2024-05-08T20:10:14.219 INFO:tasks.workunit.client.1.smithi188.stdout: 1 65805 9.75 MB/sec warmup 38 sec latency 4.529 ms 2024-05-08T20:10:14.551 INFO:tasks.workunit.client.0.smithi083.stdout: 1 107127 15.66 MB/sec warmup 38 sec latency 75.122 ms 2024-05-08T20:10:15.219 INFO:tasks.workunit.client.1.smithi188.stdout: 1 67675 9.85 MB/sec warmup 39 sec latency 4.399 ms 2024-05-08T20:10:15.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:14 smithi188 ceph-mon[48761]: pgmap v325: 129 pgs: 129 active+clean; 47 MiB data, 461 MiB used, 715 GiB / 715 GiB avail; 9.4 KiB/s rd, 4.1 MiB/s wr, 209 op/s 2024-05-08T20:10:15.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:14 smithi083 ceph-mon[38695]: pgmap v325: 129 pgs: 129 active+clean; 47 MiB data, 461 MiB used, 715 GiB / 715 GiB avail; 9.4 KiB/s rd, 4.1 MiB/s wr, 209 op/s 2024-05-08T20:10:15.551 INFO:tasks.workunit.client.0.smithi083.stdout: 1 109372 15.57 MB/sec warmup 39 sec latency 74.492 ms 2024-05-08T20:10:16.219 INFO:tasks.workunit.client.1.smithi188.stdout: 1 69281 9.74 MB/sec warmup 40 sec latency 8.116 ms 2024-05-08T20:10:16.551 INFO:tasks.workunit.client.0.smithi083.stdout: 1 112417 15.52 MB/sec warmup 40 sec latency 52.752 ms 2024-05-08T20:10:17.219 INFO:tasks.workunit.client.1.smithi188.stdout: 1 71200 9.83 MB/sec warmup 41 sec latency 7.265 ms 2024-05-08T20:10:17.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:16 smithi188 ceph-mon[48761]: pgmap v326: 129 pgs: 129 active+clean; 50 MiB data, 457 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.7 MiB/s wr, 214 op/s 2024-05-08T20:10:17.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:16 smithi083 ceph-mon[38695]: pgmap v326: 129 pgs: 129 active+clean; 50 MiB data, 457 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.7 MiB/s wr, 214 op/s 2024-05-08T20:10:17.551 INFO:tasks.workunit.client.0.smithi083.stdout: 1 115434 15.57 MB/sec warmup 41 sec latency 52.888 ms 2024-05-08T20:10:18.219 INFO:tasks.workunit.client.1.smithi188.stdout: 1 72784 9.73 MB/sec warmup 42 sec latency 8.488 ms 2024-05-08T20:10:18.551 INFO:tasks.workunit.client.0.smithi083.stdout: 1 118016 15.54 MB/sec warmup 42 sec latency 58.422 ms 2024-05-08T20:10:19.219 INFO:tasks.workunit.client.1.smithi188.stdout: 1 74704 9.80 MB/sec warmup 43 sec latency 3.989 ms 2024-05-08T20:10:19.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:18 smithi188 ceph-mon[48761]: pgmap v327: 129 pgs: 129 active+clean; 50 MiB data, 457 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.9 MiB/s wr, 193 op/s 2024-05-08T20:10:19.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:18 smithi083 ceph-mon[38695]: pgmap v327: 129 pgs: 129 active+clean; 50 MiB data, 457 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.9 MiB/s wr, 193 op/s 2024-05-08T20:10:19.551 INFO:tasks.workunit.client.0.smithi083.stdout: 1 121443 15.62 MB/sec warmup 43 sec latency 71.947 ms 2024-05-08T20:10:20.219 INFO:tasks.workunit.client.1.smithi188.stdout: 1 76258 9.72 MB/sec warmup 44 sec latency 8.256 ms 2024-05-08T20:10:20.551 INFO:tasks.workunit.client.0.smithi083.stdout: 1 124983 15.70 MB/sec warmup 44 sec latency 58.445 ms 2024-05-08T20:10:21.219 INFO:tasks.workunit.client.1.smithi188.stdout: 1 78177 9.78 MB/sec warmup 45 sec latency 6.270 ms 2024-05-08T20:10:21.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:20 smithi188 ceph-mon[48761]: pgmap v328: 129 pgs: 129 active+clean; 52 MiB data, 457 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.8 MiB/s wr, 242 op/s 2024-05-08T20:10:21.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:20 smithi083 ceph-mon[38695]: pgmap v328: 129 pgs: 129 active+clean; 52 MiB data, 457 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.8 MiB/s wr, 242 op/s 2024-05-08T20:10:21.551 INFO:tasks.workunit.client.0.smithi083.stdout: 1 128459 15.77 MB/sec warmup 45 sec latency 52.471 ms 2024-05-08T20:10:22.220 INFO:tasks.workunit.client.1.smithi188.stdout: 1 79706 9.72 MB/sec warmup 46 sec latency 6.057 ms 2024-05-08T20:10:22.551 INFO:tasks.workunit.client.0.smithi083.stdout: 1 131955 15.84 MB/sec warmup 46 sec latency 58.417 ms 2024-05-08T20:10:23.220 INFO:tasks.workunit.client.1.smithi188.stdout: 1 81637 9.77 MB/sec warmup 47 sec latency 7.879 ms 2024-05-08T20:10:23.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:22 smithi188 ceph-mon[48761]: pgmap v329: 129 pgs: 129 active+clean; 53 MiB data, 468 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 193 op/s 2024-05-08T20:10:23.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:22 smithi083 ceph-mon[38695]: pgmap v329: 129 pgs: 129 active+clean; 53 MiB data, 468 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 193 op/s 2024-05-08T20:10:23.552 INFO:tasks.workunit.client.0.smithi083.stdout: 1 134570 15.81 MB/sec warmup 47 sec latency 66.210 ms 2024-05-08T20:10:24.220 INFO:tasks.workunit.client.1.smithi188.stdout: 1 83142 9.71 MB/sec warmup 48 sec latency 4.116 ms 2024-05-08T20:10:24.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:10:24.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:10:24.552 INFO:tasks.workunit.client.0.smithi083.stdout: 1 137238 15.73 MB/sec warmup 48 sec latency 50.609 ms 2024-05-08T20:10:25.220 INFO:tasks.workunit.client.1.smithi188.stdout: 1 84975 9.70 MB/sec warmup 49 sec latency 8.073 ms 2024-05-08T20:10:25.337 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:24 smithi188 ceph-mon[48761]: pgmap v330: 129 pgs: 129 active+clean; 54 MiB data, 476 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.1 MiB/s wr, 186 op/s 2024-05-08T20:10:25.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:24 smithi083 ceph-mon[38695]: pgmap v330: 129 pgs: 129 active+clean; 54 MiB data, 476 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.1 MiB/s wr, 186 op/s 2024-05-08T20:10:25.552 INFO:tasks.workunit.client.0.smithi083.stdout: 1 140007 15.76 MB/sec warmup 49 sec latency 50.108 ms 2024-05-08T20:10:26.220 INFO:tasks.workunit.client.1.smithi188.stdout: 1 86579 9.70 MB/sec warmup 50 sec latency 6.624 ms 2024-05-08T20:10:26.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:26 smithi083 ceph-mon[38695]: pgmap v331: 129 pgs: 129 active+clean; 55 MiB data, 488 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.7 MiB/s wr, 229 op/s 2024-05-08T20:10:26.552 INFO:tasks.workunit.client.0.smithi083.stdout: 1 142798 15.73 MB/sec warmup 50 sec latency 59.455 ms 2024-05-08T20:10:26.587 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:26 smithi188 ceph-mon[48761]: pgmap v331: 129 pgs: 129 active+clean; 55 MiB data, 488 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.7 MiB/s wr, 229 op/s 2024-05-08T20:10:27.220 INFO:tasks.workunit.client.1.smithi188.stdout: 1 88493 9.69 MB/sec warmup 51 sec latency 7.320 ms 2024-05-08T20:10:27.552 INFO:tasks.workunit.client.0.smithi083.stdout: 1 146204 15.79 MB/sec warmup 51 sec latency 59.608 ms 2024-05-08T20:10:28.220 INFO:tasks.workunit.client.1.smithi188.stdout: 1 89960 9.64 MB/sec warmup 52 sec latency 9.153 ms 2024-05-08T20:10:28.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:28 smithi083 ceph-mon[38695]: pgmap v332: 129 pgs: 129 active+clean; 54 MiB data, 488 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.7 MiB/s wr, 176 op/s 2024-05-08T20:10:28.552 INFO:tasks.workunit.client.0.smithi083.stdout: 1 149733 15.86 MB/sec warmup 52 sec latency 59.560 ms 2024-05-08T20:10:28.587 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:28 smithi188 ceph-mon[48761]: pgmap v332: 129 pgs: 129 active+clean; 54 MiB data, 488 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.7 MiB/s wr, 176 op/s 2024-05-08T20:10:29.220 INFO:tasks.workunit.client.1.smithi188.stdout: 1 92013 9.69 MB/sec warmup 53 sec latency 4.778 ms 2024-05-08T20:10:29.552 INFO:tasks.workunit.client.0.smithi083.stdout: 1 153227 15.91 MB/sec warmup 53 sec latency 57.093 ms 2024-05-08T20:10:30.220 INFO:tasks.workunit.client.1.smithi188.stdout: 1 93439 9.63 MB/sec warmup 54 sec latency 9.236 ms 2024-05-08T20:10:30.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:30 smithi083 ceph-mon[38695]: pgmap v333: 129 pgs: 129 active+clean; 59 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.8 MiB/s wr, 234 op/s 2024-05-08T20:10:30.552 INFO:tasks.workunit.client.0.smithi083.stdout: 1 155842 15.88 MB/sec warmup 54 sec latency 56.964 ms 2024-05-08T20:10:30.587 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:30 smithi188 ceph-mon[48761]: pgmap v333: 129 pgs: 129 active+clean; 59 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.8 MiB/s wr, 234 op/s 2024-05-08T20:10:31.220 INFO:tasks.workunit.client.1.smithi188.stdout: 1 95547 9.68 MB/sec warmup 55 sec latency 8.048 ms 2024-05-08T20:10:31.552 INFO:tasks.workunit.client.0.smithi083.stdout: 1 159012 15.91 MB/sec warmup 55 sec latency 48.984 ms 2024-05-08T20:10:32.220 INFO:tasks.workunit.client.1.smithi188.stdout: 1 96931 9.63 MB/sec warmup 56 sec latency 11.588 ms 2024-05-08T20:10:32.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:32 smithi083 ceph-mon[38695]: pgmap v334: 129 pgs: 129 active+clean; 59 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.2 MiB/s wr, 188 op/s 2024-05-08T20:10:32.552 INFO:tasks.workunit.client.0.smithi083.stdout: 1 160447 15.75 MB/sec warmup 56 sec latency 327.280 ms 2024-05-08T20:10:32.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:32 smithi188 ceph-mon[48761]: pgmap v334: 129 pgs: 129 active+clean; 59 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.2 MiB/s wr, 188 op/s 2024-05-08T20:10:33.220 INFO:tasks.workunit.client.1.smithi188.stdout: 1 98827 9.65 MB/sec warmup 57 sec latency 27.075 ms 2024-05-08T20:10:33.552 INFO:tasks.workunit.client.0.smithi083.stdout: 1 160523 15.48 MB/sec warmup 57 sec latency 400.233 ms 2024-05-08T20:10:34.220 INFO:tasks.workunit.client.1.smithi188.stdout: 1 100336 9.61 MB/sec warmup 58 sec latency 12.851 ms 2024-05-08T20:10:34.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:34 smithi083 ceph-mon[38695]: pgmap v335: 129 pgs: 129 active+clean; 58 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 3.9 MiB/s wr, 184 op/s 2024-05-08T20:10:34.552 INFO:tasks.workunit.client.0.smithi083.stdout: 1 163872 15.53 MB/sec warmup 58 sec latency 416.534 ms 2024-05-08T20:10:34.587 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:34 smithi188 ceph-mon[48761]: pgmap v335: 129 pgs: 129 active+clean; 58 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 3.9 MiB/s wr, 184 op/s 2024-05-08T20:10:35.220 INFO:tasks.workunit.client.1.smithi188.stdout: 1 102259 9.63 MB/sec warmup 59 sec latency 8.909 ms 2024-05-08T20:10:35.552 INFO:tasks.workunit.client.0.smithi083.stdout: 1 167416 15.59 MB/sec warmup 59 sec latency 58.388 ms 2024-05-08T20:10:36.220 INFO:tasks.workunit.client.1.smithi188.stdout: 1 103812 9.61 MB/sec warmup 60 sec latency 9.730 ms 2024-05-08T20:10:36.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:36 smithi083 ceph-mon[38695]: pgmap v336: 129 pgs: 129 active+clean; 61 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.7 MiB/s wr, 240 op/s 2024-05-08T20:10:36.552 INFO:tasks.workunit.client.0.smithi083.stdout: 1 170057 15.57 MB/sec warmup 60 sec latency 56.286 ms 2024-05-08T20:10:36.587 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:36 smithi188 ceph-mon[48761]: pgmap v336: 129 pgs: 129 active+clean; 61 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.7 MiB/s wr, 240 op/s 2024-05-08T20:10:37.220 INFO:tasks.workunit.client.1.smithi188.stdout: 1 105668 9.62 MB/sec warmup 61 sec latency 4.807 ms 2024-05-08T20:10:37.553 INFO:tasks.workunit.client.0.smithi083.stdout: 1 173254 15.60 MB/sec warmup 61 sec latency 58.275 ms 2024-05-08T20:10:38.220 INFO:tasks.workunit.client.1.smithi188.stdout: 1 107242 9.60 MB/sec warmup 62 sec latency 7.707 ms 2024-05-08T20:10:38.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:38 smithi083 ceph-mon[38695]: pgmap v337: 129 pgs: 129 active+clean; 63 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.0 MiB/s wr, 180 op/s 2024-05-08T20:10:38.552 INFO:tasks.workunit.client.0.smithi083.stdout: 1 176168 15.56 MB/sec warmup 62 sec latency 58.420 ms 2024-05-08T20:10:38.587 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:38 smithi188 ceph-mon[48761]: pgmap v337: 129 pgs: 129 active+clean; 63 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.0 MiB/s wr, 180 op/s 2024-05-08T20:10:39.220 INFO:tasks.workunit.client.1.smithi188.stdout: 1 109030 9.57 MB/sec warmup 63 sec latency 4.849 ms 2024-05-08T20:10:39.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:10:39.552 INFO:tasks.workunit.client.0.smithi083.stdout: 1 178853 15.59 MB/sec warmup 63 sec latency 57.304 ms 2024-05-08T20:10:39.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:10:40.220 INFO:tasks.workunit.client.1.smithi188.stdout: 1 110674 9.59 MB/sec warmup 64 sec latency 14.874 ms 2024-05-08T20:10:40.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:40 smithi083 ceph-mon[38695]: pgmap v338: 129 pgs: 129 active+clean; 64 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 9.3 KiB/s rd, 4.8 MiB/s wr, 227 op/s 2024-05-08T20:10:40.552 INFO:tasks.workunit.client.0.smithi083.stdout: 1 181902 15.58 MB/sec warmup 64 sec latency 60.343 ms 2024-05-08T20:10:40.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:40 smithi188 ceph-mon[48761]: pgmap v338: 129 pgs: 129 active+clean; 64 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 9.3 KiB/s rd, 4.8 MiB/s wr, 227 op/s 2024-05-08T20:10:41.220 INFO:tasks.workunit.client.1.smithi188.stdout: 1 112445 9.55 MB/sec warmup 65 sec latency 8.880 ms 2024-05-08T20:10:41.553 INFO:tasks.workunit.client.0.smithi083.stdout: 1 185379 15.63 MB/sec warmup 65 sec latency 59.874 ms 2024-05-08T20:10:42.220 INFO:tasks.workunit.client.1.smithi188.stdout: 1 114085 9.59 MB/sec warmup 66 sec latency 6.856 ms 2024-05-08T20:10:42.438 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:42 smithi083 ceph-mon[38695]: pgmap v339: 129 pgs: 129 active+clean; 63 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.8 MiB/s wr, 170 op/s 2024-05-08T20:10:42.553 INFO:tasks.workunit.client.0.smithi083.stdout: 1 188892 15.68 MB/sec warmup 66 sec latency 57.982 ms 2024-05-08T20:10:42.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:42 smithi188 ceph-mon[48761]: pgmap v339: 129 pgs: 129 active+clean; 63 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.8 MiB/s wr, 170 op/s 2024-05-08T20:10:43.220 INFO:tasks.workunit.client.1.smithi188.stdout: 1 115909 9.55 MB/sec warmup 67 sec latency 7.240 ms 2024-05-08T20:10:43.553 INFO:tasks.workunit.client.0.smithi083.stdout: 1 192352 15.73 MB/sec warmup 67 sec latency 58.587 ms 2024-05-08T20:10:44.220 INFO:tasks.workunit.client.1.smithi188.stdout: 1 117563 9.59 MB/sec warmup 68 sec latency 8.697 ms 2024-05-08T20:10:44.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:44 smithi083 ceph-mon[38695]: pgmap v340: 129 pgs: 129 active+clean; 67 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.0 MiB/s wr, 222 op/s 2024-05-08T20:10:44.553 INFO:tasks.workunit.client.0.smithi083.stdout: 1 194742 15.69 MB/sec warmup 68 sec latency 308.681 ms 2024-05-08T20:10:44.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:44 smithi188 ceph-mon[48761]: pgmap v340: 129 pgs: 129 active+clean; 67 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.0 MiB/s wr, 222 op/s 2024-05-08T20:10:45.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 119294 9.53 MB/sec warmup 69 sec latency 5.090 ms 2024-05-08T20:10:45.553 INFO:tasks.workunit.client.0.smithi083.stdout: 1 197574 15.66 MB/sec warmup 69 sec latency 58.534 ms 2024-05-08T20:10:46.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 121044 9.59 MB/sec warmup 70 sec latency 14.243 ms 2024-05-08T20:10:46.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:46 smithi083 ceph-mon[38695]: pgmap v341: 129 pgs: 129 active+clean; 67 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.7 MiB/s wr, 229 op/s 2024-05-08T20:10:46.553 INFO:tasks.workunit.client.0.smithi083.stdout: 1 200534 15.68 MB/sec warmup 70 sec latency 50.202 ms 2024-05-08T20:10:46.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:46 smithi188 ceph-mon[48761]: pgmap v341: 129 pgs: 129 active+clean; 67 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.7 MiB/s wr, 229 op/s 2024-05-08T20:10:47.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 122713 9.53 MB/sec warmup 71 sec latency 5.806 ms 2024-05-08T20:10:47.553 INFO:tasks.workunit.client.0.smithi083.stdout: 1 203178 15.66 MB/sec warmup 71 sec latency 49.702 ms 2024-05-08T20:10:48.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 124501 9.58 MB/sec warmup 72 sec latency 7.408 ms 2024-05-08T20:10:48.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:48 smithi083 ceph-mon[38695]: pgmap v342: 129 pgs: 129 active+clean; 68 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 173 op/s 2024-05-08T20:10:48.553 INFO:tasks.workunit.client.0.smithi083.stdout: 1 206656 15.71 MB/sec warmup 72 sec latency 58.177 ms 2024-05-08T20:10:48.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:48 smithi188 ceph-mon[48761]: pgmap v342: 129 pgs: 129 active+clean; 68 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 173 op/s 2024-05-08T20:10:49.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 126112 9.53 MB/sec warmup 73 sec latency 4.753 ms 2024-05-08T20:10:49.553 INFO:tasks.workunit.client.0.smithi083.stdout: 1 210086 15.75 MB/sec warmup 73 sec latency 88.722 ms 2024-05-08T20:10:50.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 127983 9.58 MB/sec warmup 74 sec latency 10.451 ms 2024-05-08T20:10:50.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:50 smithi083 ceph-mon[38695]: pgmap v343: 129 pgs: 129 active+clean; 73 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 9.4 KiB/s rd, 4.8 MiB/s wr, 234 op/s 2024-05-08T20:10:50.553 INFO:tasks.workunit.client.0.smithi083.stdout: 1 213619 15.79 MB/sec warmup 74 sec latency 95.201 ms 2024-05-08T20:10:50.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:50 smithi188 ceph-mon[48761]: pgmap v343: 129 pgs: 129 active+clean; 73 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 9.4 KiB/s rd, 4.8 MiB/s wr, 234 op/s 2024-05-08T20:10:51.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 129563 9.53 MB/sec warmup 75 sec latency 5.700 ms 2024-05-08T20:10:51.553 INFO:tasks.workunit.client.0.smithi083.stdout: 1 217160 15.84 MB/sec warmup 75 sec latency 99.848 ms 2024-05-08T20:10:52.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 131438 9.57 MB/sec warmup 76 sec latency 6.630 ms 2024-05-08T20:10:52.437 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:52 smithi083 ceph-mon[38695]: pgmap v344: 129 pgs: 129 active+clean; 72 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.1 MiB/s wr, 189 op/s 2024-05-08T20:10:52.553 INFO:tasks.workunit.client.0.smithi083.stdout: 1 220698 15.88 MB/sec warmup 76 sec latency 55.525 ms 2024-05-08T20:10:52.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:52 smithi188 ceph-mon[48761]: pgmap v344: 129 pgs: 129 active+clean; 72 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.1 MiB/s wr, 189 op/s 2024-05-08T20:10:53.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 132976 9.53 MB/sec warmup 77 sec latency 12.747 ms 2024-05-08T20:10:53.553 INFO:tasks.workunit.client.0.smithi083.stdout: 1 224195 15.92 MB/sec warmup 77 sec latency 50.112 ms 2024-05-08T20:10:54.187 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:53 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:10:54.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 134892 9.57 MB/sec warmup 78 sec latency 5.508 ms 2024-05-08T20:10:54.338 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:53 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:10:54.553 INFO:tasks.workunit.client.0.smithi083.stdout: 1 226815 15.90 MB/sec warmup 78 sec latency 46.944 ms 2024-05-08T20:10:55.188 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:54 smithi083 ceph-mon[38695]: pgmap v345: 129 pgs: 129 active+clean; 72 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.9 MiB/s wr, 194 op/s 2024-05-08T20:10:55.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 136406 9.53 MB/sec warmup 79 sec latency 11.056 ms 2024-05-08T20:10:55.338 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:54 smithi188 ceph-mon[48761]: pgmap v345: 129 pgs: 129 active+clean; 72 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.9 MiB/s wr, 194 op/s 2024-05-08T20:10:55.553 INFO:tasks.workunit.client.0.smithi083.stdout: 1 229975 15.92 MB/sec warmup 79 sec latency 55.992 ms 2024-05-08T20:10:56.188 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:55 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:10:56.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 138155 9.53 MB/sec warmup 80 sec latency 5.902 ms 2024-05-08T20:10:56.338 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:55 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:10:56.553 INFO:tasks.workunit.client.0.smithi083.stdout: 1 232497 15.87 MB/sec warmup 80 sec latency 55.892 ms 2024-05-08T20:10:57.188 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:56 smithi083 ceph-mon[38695]: pgmap v346: 129 pgs: 129 active+clean; 74 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.4 MiB/s wr, 192 op/s 2024-05-08T20:10:57.188 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:56 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:10:57.188 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:56 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:10:57.189 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:56 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:10:57.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 139658 9.49 MB/sec warmup 81 sec latency 11.995 ms 2024-05-08T20:10:57.338 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:56 smithi188 ceph-mon[48761]: pgmap v346: 129 pgs: 129 active+clean; 74 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.4 MiB/s wr, 192 op/s 2024-05-08T20:10:57.338 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:56 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:10:57.338 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:56 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:10:57.338 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:56 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:10:57.553 INFO:tasks.workunit.client.0.smithi083.stdout: 1 235635 15.90 MB/sec warmup 81 sec latency 58.404 ms 2024-05-08T20:10:58.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 141668 9.53 MB/sec warmup 82 sec latency 6.542 ms 2024-05-08T20:10:58.554 INFO:tasks.workunit.client.0.smithi083.stdout: 1 238509 15.89 MB/sec warmup 82 sec latency 58.412 ms 2024-05-08T20:10:59.088 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:10:58 smithi188 ceph-mon[48761]: pgmap v347: 129 pgs: 129 active+clean; 75 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 182 op/s 2024-05-08T20:10:59.188 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:10:58 smithi083 ceph-mon[38695]: pgmap v347: 129 pgs: 129 active+clean; 75 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 182 op/s 2024-05-08T20:10:59.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 143075 9.49 MB/sec warmup 83 sec latency 18.123 ms 2024-05-08T20:10:59.554 INFO:tasks.workunit.client.0.smithi083.stdout: 1 241983 15.92 MB/sec warmup 83 sec latency 55.642 ms 2024-05-08T20:11:00.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 145076 9.51 MB/sec warmup 84 sec latency 4.532 ms 2024-05-08T20:11:00.338 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:00 smithi188 ceph-mon[48761]: pgmap v348: 129 pgs: 129 active+clean; 75 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.5 MiB/s wr, 226 op/s 2024-05-08T20:11:00.438 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:00 smithi083 ceph-mon[38695]: pgmap v348: 129 pgs: 129 active+clean; 75 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.5 MiB/s wr, 226 op/s 2024-05-08T20:11:00.554 INFO:tasks.workunit.client.0.smithi083.stdout: 1 245472 15.96 MB/sec warmup 84 sec latency 55.883 ms 2024-05-08T20:11:01.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 146418 9.48 MB/sec warmup 85 sec latency 16.549 ms 2024-05-08T20:11:01.554 INFO:tasks.workunit.client.0.smithi083.stdout: 1 248389 15.98 MB/sec warmup 85 sec latency 67.049 ms 2024-05-08T20:11:02.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 148353 9.50 MB/sec warmup 86 sec latency 4.757 ms 2024-05-08T20:11:02.438 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:02 smithi083 ceph-mon[38695]: pgmap v349: 129 pgs: 129 active+clean; 76 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.7 MiB/s wr, 167 op/s 2024-05-08T20:11:02.554 INFO:tasks.workunit.client.0.smithi083.stdout: 1 251308 15.95 MB/sec warmup 86 sec latency 58.488 ms 2024-05-08T20:11:02.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:02 smithi188 ceph-mon[48761]: pgmap v349: 129 pgs: 129 active+clean; 76 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.7 MiB/s wr, 167 op/s 2024-05-08T20:11:03.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 149884 9.48 MB/sec warmup 87 sec latency 5.063 ms 2024-05-08T20:11:03.554 INFO:tasks.workunit.client.0.smithi083.stdout: 1 253774 15.91 MB/sec warmup 87 sec latency 53.655 ms 2024-05-08T20:11:04.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 151703 9.49 MB/sec warmup 88 sec latency 11.316 ms 2024-05-08T20:11:04.438 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:04 smithi083 ceph-mon[38695]: pgmap v350: 129 pgs: 129 active+clean; 79 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.9 MiB/s wr, 211 op/s 2024-05-08T20:11:04.554 INFO:tasks.workunit.client.0.smithi083.stdout: 1 256410 15.89 MB/sec warmup 88 sec latency 58.293 ms 2024-05-08T20:11:04.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:04 smithi188 ceph-mon[48761]: pgmap v350: 129 pgs: 129 active+clean; 79 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.9 MiB/s wr, 211 op/s 2024-05-08T20:11:05.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 153306 9.48 MB/sec warmup 89 sec latency 13.182 ms 2024-05-08T20:11:05.554 INFO:tasks.workunit.client.0.smithi083.stdout: 1 258419 15.85 MB/sec warmup 89 sec latency 354.915 ms 2024-05-08T20:11:06.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 155100 9.46 MB/sec warmup 90 sec latency 7.262 ms 2024-05-08T20:11:06.438 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:06 smithi083 ceph-mon[38695]: pgmap v351: 129 pgs: 129 active+clean; 80 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 218 op/s 2024-05-08T20:11:06.554 INFO:tasks.workunit.client.0.smithi083.stdout: 1 259658 15.75 MB/sec warmup 90 sec latency 387.096 ms 2024-05-08T20:11:06.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:06 smithi188 ceph-mon[48761]: pgmap v351: 129 pgs: 129 active+clean; 80 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 218 op/s 2024-05-08T20:11:07.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 156714 9.48 MB/sec warmup 91 sec latency 9.603 ms 2024-05-08T20:11:07.554 INFO:tasks.workunit.client.0.smithi083.stdout: 1 263189 15.78 MB/sec warmup 91 sec latency 58.308 ms 2024-05-08T20:11:08.222 INFO:tasks.workunit.client.1.smithi188.stdout: 1 158513 9.45 MB/sec warmup 92 sec latency 9.196 ms 2024-05-08T20:11:08.554 INFO:tasks.workunit.client.0.smithi083.stdout: 1 265706 15.75 MB/sec warmup 92 sec latency 58.271 ms 2024-05-08T20:11:09.088 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:08 smithi188 ceph-mon[48761]: pgmap v352: 129 pgs: 129 active+clean; 79 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.7 MiB/s wr, 162 op/s 2024-05-08T20:11:09.180 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:08 smithi083 ceph-mon[38695]: pgmap v352: 129 pgs: 129 active+clean; 79 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.7 MiB/s wr, 162 op/s 2024-05-08T20:11:09.221 INFO:tasks.workunit.client.1.smithi188.stdout: 1 160167 9.48 MB/sec warmup 93 sec latency 9.965 ms 2024-05-08T20:11:09.554 INFO:tasks.workunit.client.0.smithi083.stdout: 1 268438 15.72 MB/sec warmup 93 sec latency 58.463 ms 2024-05-08T20:11:10.088 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:11:10.188 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:11:10.222 INFO:tasks.workunit.client.1.smithi188.stdout: 1 161970 9.45 MB/sec warmup 94 sec latency 6.897 ms 2024-05-08T20:11:10.554 INFO:tasks.workunit.client.0.smithi083.stdout: 1 271090 15.74 MB/sec warmup 94 sec latency 58.323 ms 2024-05-08T20:11:11.089 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:10 smithi188 ceph-mon[48761]: pgmap v353: 129 pgs: 129 active+clean; 82 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.6 MiB/s wr, 222 op/s 2024-05-08T20:11:11.188 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:10 smithi083 ceph-mon[38695]: pgmap v353: 129 pgs: 129 active+clean; 82 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.6 MiB/s wr, 222 op/s 2024-05-08T20:11:11.222 INFO:tasks.workunit.client.1.smithi188.stdout: 1 163628 9.48 MB/sec warmup 95 sec latency 7.052 ms 2024-05-08T20:11:11.554 INFO:tasks.workunit.client.0.smithi083.stdout: 1 274078 15.73 MB/sec warmup 95 sec latency 59.561 ms 2024-05-08T20:11:12.222 INFO:tasks.workunit.client.1.smithi188.stdout: 1 165327 9.44 MB/sec warmup 96 sec latency 10.696 ms 2024-05-08T20:11:12.438 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:12 smithi083 ceph-mon[38695]: pgmap v354: 129 pgs: 129 active+clean; 83 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 188 op/s 2024-05-08T20:11:12.554 INFO:tasks.workunit.client.0.smithi083.stdout: 1 277499 15.76 MB/sec warmup 96 sec latency 58.080 ms 2024-05-08T20:11:12.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:12 smithi188 ceph-mon[48761]: pgmap v354: 129 pgs: 129 active+clean; 83 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 188 op/s 2024-05-08T20:11:13.222 INFO:tasks.workunit.client.1.smithi188.stdout: 1 167086 9.48 MB/sec warmup 97 sec latency 6.364 ms 2024-05-08T20:11:13.554 INFO:tasks.workunit.client.0.smithi083.stdout: 1 280982 15.79 MB/sec warmup 97 sec latency 58.662 ms 2024-05-08T20:11:14.222 INFO:tasks.workunit.client.1.smithi188.stdout: 1 168719 9.44 MB/sec warmup 98 sec latency 13.326 ms 2024-05-08T20:11:14.438 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:14 smithi083 ceph-mon[38695]: pgmap v355: 129 pgs: 129 active+clean; 85 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 9.3 KiB/s rd, 3.9 MiB/s wr, 200 op/s 2024-05-08T20:11:14.554 INFO:tasks.workunit.client.0.smithi083.stdout: 1 284468 15.83 MB/sec warmup 98 sec latency 58.097 ms 2024-05-08T20:11:14.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:14 smithi188 ceph-mon[48761]: pgmap v355: 129 pgs: 129 active+clean; 85 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 9.3 KiB/s rd, 3.9 MiB/s wr, 200 op/s 2024-05-08T20:11:15.222 INFO:tasks.workunit.client.1.smithi188.stdout: 1 170518 9.48 MB/sec warmup 99 sec latency 11.051 ms 2024-05-08T20:11:15.554 INFO:tasks.workunit.client.0.smithi083.stdout: 1 287111 15.81 MB/sec warmup 99 sec latency 44.183 ms 2024-05-08T20:11:16.222 INFO:tasks.workunit.client.1.smithi188.stdout: 1 172101 9.44 MB/sec warmup 100 sec latency 13.690 ms 2024-05-08T20:11:16.554 INFO:tasks.workunit.client.0.smithi083.stdout: 1 290252 15.82 MB/sec warmup 100 sec latency 61.489 ms 2024-05-08T20:11:16.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:16 smithi188 ceph-mon[48761]: pgmap v356: 129 pgs: 129 active+clean; 87 MiB data, 681 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.5 MiB/s wr, 203 op/s 2024-05-08T20:11:16.688 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:16 smithi083 ceph-mon[38695]: pgmap v356: 129 pgs: 129 active+clean; 87 MiB data, 681 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.5 MiB/s wr, 203 op/s 2024-05-08T20:11:17.222 INFO:tasks.workunit.client.1.smithi188.stdout: 1 173978 9.47 MB/sec warmup 101 sec latency 5.828 ms 2024-05-08T20:11:17.554 INFO:tasks.workunit.client.0.smithi083.stdout: 1 292363 15.78 MB/sec warmup 101 sec latency 190.997 ms 2024-05-08T20:11:18.222 INFO:tasks.workunit.client.1.smithi188.stdout: 1 175490 9.44 MB/sec warmup 102 sec latency 9.277 ms 2024-05-08T20:11:18.438 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:18 smithi083 ceph-mon[38695]: pgmap v357: 129 pgs: 129 active+clean; 88 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.0 MiB/s wr, 193 op/s 2024-05-08T20:11:18.554 INFO:tasks.workunit.client.0.smithi083.stdout: 1 295348 15.77 MB/sec warmup 102 sec latency 57.408 ms 2024-05-08T20:11:18.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:18 smithi188 ceph-mon[48761]: pgmap v357: 129 pgs: 129 active+clean; 88 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.0 MiB/s wr, 193 op/s 2024-05-08T20:11:19.222 INFO:tasks.workunit.client.1.smithi188.stdout: 1 177427 9.47 MB/sec warmup 103 sec latency 6.464 ms 2024-05-08T20:11:19.554 INFO:tasks.workunit.client.0.smithi083.stdout: 1 298786 15.80 MB/sec warmup 103 sec latency 60.972 ms 2024-05-08T20:11:20.222 INFO:tasks.workunit.client.1.smithi188.stdout: 1 178949 9.45 MB/sec warmup 104 sec latency 9.947 ms 2024-05-08T20:11:20.555 INFO:tasks.workunit.client.0.smithi083.stdout: 1 302325 15.84 MB/sec warmup 104 sec latency 58.042 ms 2024-05-08T20:11:20.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:20 smithi188 ceph-mon[48761]: pgmap v358: 129 pgs: 129 active+clean; 90 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.8 MiB/s wr, 235 op/s 2024-05-08T20:11:20.688 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:20 smithi083 ceph-mon[38695]: pgmap v358: 129 pgs: 129 active+clean; 90 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.8 MiB/s wr, 235 op/s 2024-05-08T20:11:21.222 INFO:tasks.workunit.client.1.smithi188.stdout: 1 180753 9.44 MB/sec warmup 105 sec latency 9.048 ms 2024-05-08T20:11:21.555 INFO:tasks.workunit.client.0.smithi083.stdout: 1 305803 15.87 MB/sec warmup 105 sec latency 60.984 ms 2024-05-08T20:11:22.222 INFO:tasks.workunit.client.1.smithi188.stdout: 1 182365 9.45 MB/sec warmup 106 sec latency 9.866 ms 2024-05-08T20:11:22.438 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:22 smithi083 ceph-mon[38695]: pgmap v359: 129 pgs: 129 active+clean; 89 MiB data, 726 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 176 op/s 2024-05-08T20:11:22.555 INFO:tasks.workunit.client.0.smithi083.stdout: 1 309296 15.89 MB/sec warmup 106 sec latency 60.735 ms 2024-05-08T20:11:22.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:22 smithi188 ceph-mon[48761]: pgmap v359: 129 pgs: 129 active+clean; 89 MiB data, 726 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 176 op/s 2024-05-08T20:11:23.222 INFO:tasks.workunit.client.1.smithi188.stdout: 1 184237 9.45 MB/sec warmup 107 sec latency 9.903 ms 2024-05-08T20:11:23.555 INFO:tasks.workunit.client.0.smithi083.stdout: 1 312837 15.92 MB/sec warmup 107 sec latency 52.425 ms 2024-05-08T20:11:24.188 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:11:24.222 INFO:tasks.workunit.client.1.smithi188.stdout: 1 185715 9.42 MB/sec warmup 108 sec latency 18.414 ms 2024-05-08T20:11:24.338 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:11:24.555 INFO:tasks.workunit.client.0.smithi083.stdout: 1 316265 15.95 MB/sec warmup 108 sec latency 58.444 ms 2024-05-08T20:11:25.188 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:24 smithi083 ceph-mon[38695]: pgmap v360: 129 pgs: 129 active+clean; 93 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.1 MiB/s wr, 222 op/s 2024-05-08T20:11:25.222 INFO:tasks.workunit.client.1.smithi188.stdout: 1 187746 9.45 MB/sec warmup 109 sec latency 10.745 ms 2024-05-08T20:11:25.288 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:24 smithi188 ceph-mon[48761]: pgmap v360: 129 pgs: 129 active+clean; 93 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.1 MiB/s wr, 222 op/s 2024-05-08T20:11:25.555 INFO:tasks.workunit.client.0.smithi083.stdout: 1 318683 15.92 MB/sec warmup 109 sec latency 58.077 ms 2024-05-08T20:11:26.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 189126 9.42 MB/sec warmup 110 sec latency 7.004 ms 2024-05-08T20:11:26.479 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:26 smithi083 ceph-mon[38695]: pgmap v361: 129 pgs: 129 active+clean; 93 MiB data, 747 MiB used, 714 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.5 MiB/s wr, 222 op/s 2024-05-08T20:11:26.555 INFO:tasks.workunit.client.0.smithi083.stdout: 1 321140 15.89 MB/sec warmup 110 sec latency 58.481 ms 2024-05-08T20:11:26.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:26 smithi188 ceph-mon[48761]: pgmap v361: 129 pgs: 129 active+clean; 93 MiB data, 747 MiB used, 714 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.5 MiB/s wr, 222 op/s 2024-05-08T20:11:27.222 INFO:tasks.workunit.client.1.smithi188.stdout: 1 191020 9.43 MB/sec warmup 111 sec latency 7.091 ms 2024-05-08T20:11:27.555 INFO:tasks.workunit.client.0.smithi083.stdout: 1 324435 15.91 MB/sec warmup 111 sec latency 50.514 ms 2024-05-08T20:11:28.222 INFO:tasks.workunit.client.1.smithi188.stdout: 1 192493 9.41 MB/sec warmup 112 sec latency 13.591 ms 2024-05-08T20:11:28.438 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:28 smithi083 ceph-mon[38695]: pgmap v362: 129 pgs: 129 active+clean; 92 MiB data, 747 MiB used, 714 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.7 MiB/s wr, 172 op/s 2024-05-08T20:11:28.555 INFO:tasks.workunit.client.0.smithi083.stdout: 1 327264 15.90 MB/sec warmup 112 sec latency 50.871 ms 2024-05-08T20:11:28.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:28 smithi188 ceph-mon[48761]: pgmap v362: 129 pgs: 129 active+clean; 92 MiB data, 747 MiB used, 714 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.7 MiB/s wr, 172 op/s 2024-05-08T20:11:29.222 INFO:tasks.workunit.client.1.smithi188.stdout: 1 194325 9.42 MB/sec warmup 113 sec latency 9.198 ms 2024-05-08T20:11:29.555 INFO:tasks.workunit.client.0.smithi083.stdout: 1 330695 15.93 MB/sec warmup 113 sec latency 58.947 ms 2024-05-08T20:11:30.222 INFO:tasks.workunit.client.1.smithi188.stdout: 1 195886 9.41 MB/sec warmup 114 sec latency 10.580 ms 2024-05-08T20:11:30.555 INFO:tasks.workunit.client.0.smithi083.stdout: 1 334229 15.96 MB/sec warmup 114 sec latency 51.321 ms 2024-05-08T20:11:30.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:30 smithi188 ceph-mon[48761]: pgmap v363: 129 pgs: 129 active+clean; 94 MiB data, 760 MiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.5 MiB/s wr, 219 op/s 2024-05-08T20:11:30.688 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:30 smithi083 ceph-mon[38695]: pgmap v363: 129 pgs: 129 active+clean; 94 MiB data, 760 MiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.5 MiB/s wr, 219 op/s 2024-05-08T20:11:31.222 INFO:tasks.workunit.client.1.smithi188.stdout: 1 197646 9.40 MB/sec warmup 115 sec latency 6.750 ms 2024-05-08T20:11:31.555 INFO:tasks.workunit.client.0.smithi083.stdout: 1 337709 15.98 MB/sec warmup 115 sec latency 59.305 ms 2024-05-08T20:11:32.223 INFO:tasks.workunit.client.1.smithi188.stdout: 1 199295 9.41 MB/sec warmup 116 sec latency 9.228 ms 2024-05-08T20:11:32.555 INFO:tasks.workunit.client.0.smithi083.stdout: 1 341204 16.01 MB/sec warmup 116 sec latency 50.638 ms 2024-05-08T20:11:32.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:32 smithi188 ceph-mon[48761]: pgmap v364: 129 pgs: 129 active+clean; 94 MiB data, 754 MiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 180 op/s 2024-05-08T20:11:32.688 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:32 smithi083 ceph-mon[38695]: pgmap v364: 129 pgs: 129 active+clean; 94 MiB data, 754 MiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 180 op/s 2024-05-08T20:11:33.223 INFO:tasks.workunit.client.1.smithi188.stdout: 1 201069 9.39 MB/sec warmup 117 sec latency 9.609 ms 2024-05-08T20:11:33.555 INFO:tasks.workunit.client.0.smithi083.stdout: 1 344746 16.03 MB/sec warmup 117 sec latency 56.386 ms 2024-05-08T20:11:34.223 INFO:tasks.workunit.client.1.smithi188.stdout: 1 202689 9.41 MB/sec warmup 118 sec latency 7.946 ms 2024-05-08T20:11:34.504 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:34 smithi083 ceph-mon[38695]: pgmap v365: 129 pgs: 129 active+clean; 97 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.1 MiB/s wr, 224 op/s 2024-05-08T20:11:34.555 INFO:tasks.workunit.client.0.smithi083.stdout: 1 347368 16.02 MB/sec warmup 118 sec latency 49.309 ms 2024-05-08T20:11:34.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:34 smithi188 ceph-mon[48761]: pgmap v365: 129 pgs: 129 active+clean; 97 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.1 MiB/s wr, 224 op/s 2024-05-08T20:11:35.223 INFO:tasks.workunit.client.1.smithi188.stdout: 1 204482 9.39 MB/sec warmup 119 sec latency 7.711 ms 2024-05-08T20:11:35.555 INFO:tasks.workunit.client.0.smithi083.stdout: 1 350589 16.03 MB/sec warmup 119 sec latency 58.338 ms 2024-05-08T20:11:36.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:36 smithi188 ceph-mon[48761]: pgmap v366: 129 pgs: 129 active+clean; 97 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.2 MiB/s wr, 178 op/s 2024-05-08T20:11:36.688 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:36 smithi083 ceph-mon[38695]: pgmap v366: 129 pgs: 129 active+clean; 97 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.2 MiB/s wr, 178 op/s 2024-05-08T20:11:37.223 INFO:tasks.workunit.client.1.smithi188.stdout: 1 207828 5.71 MB/sec execute 1 sec latency 10.606 ms 2024-05-08T20:11:37.555 INFO:tasks.workunit.client.0.smithi083.stdout: 1 355440 14.05 MB/sec execute 1 sec latency 390.654 ms 2024-05-08T20:11:38.223 INFO:tasks.workunit.client.1.smithi188.stdout: 1 209615 9.55 MB/sec execute 2 sec latency 7.972 ms 2024-05-08T20:11:38.438 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:38 smithi083 ceph-mon[38695]: pgmap v367: 129 pgs: 129 active+clean; 97 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.8 MiB/s wr, 166 op/s 2024-05-08T20:11:38.556 INFO:tasks.workunit.client.0.smithi083.stdout: 1 356188 9.39 MB/sec execute 2 sec latency 467.413 ms 2024-05-08T20:11:38.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:38 smithi188 ceph-mon[48761]: pgmap v367: 129 pgs: 129 active+clean; 97 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.8 MiB/s wr, 166 op/s 2024-05-08T20:11:39.223 INFO:tasks.workunit.client.1.smithi188.stdout: 1 211260 8.21 MB/sec execute 3 sec latency 13.199 ms 2024-05-08T20:11:39.556 INFO:tasks.workunit.client.0.smithi083.stdout: 1 359176 11.21 MB/sec execute 3 sec latency 47.752 ms 2024-05-08T20:11:39.568 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:11:39.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:11:40.223 INFO:tasks.workunit.client.1.smithi188.stdout: 1 213087 9.48 MB/sec execute 4 sec latency 12.313 ms 2024-05-08T20:11:40.464 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:40 smithi188 ceph-mon[48761]: pgmap v368: 129 pgs: 129 active+clean; 100 MiB data, 828 MiB used, 714 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.7 MiB/s wr, 224 op/s 2024-05-08T20:11:40.556 INFO:tasks.workunit.client.0.smithi083.stdout: 1 362675 13.16 MB/sec execute 4 sec latency 57.900 ms 2024-05-08T20:11:40.688 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:40 smithi083 ceph-mon[38695]: pgmap v368: 129 pgs: 129 active+clean; 100 MiB data, 828 MiB used, 714 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.7 MiB/s wr, 224 op/s 2024-05-08T20:11:41.223 INFO:tasks.workunit.client.1.smithi188.stdout: 1 214648 8.73 MB/sec execute 5 sec latency 6.241 ms 2024-05-08T20:11:41.556 INFO:tasks.workunit.client.0.smithi083.stdout: 1 366145 14.31 MB/sec execute 5 sec latency 46.435 ms 2024-05-08T20:11:42.223 INFO:tasks.workunit.client.1.smithi188.stdout: 1 216559 9.35 MB/sec execute 6 sec latency 13.909 ms 2024-05-08T20:11:42.556 INFO:tasks.workunit.client.0.smithi083.stdout: 1 369664 15.10 MB/sec execute 6 sec latency 49.196 ms 2024-05-08T20:11:42.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:42 smithi188 ceph-mon[48761]: pgmap v369: 129 pgs: 129 active+clean; 101 MiB data, 835 MiB used, 714 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.9 MiB/s wr, 176 op/s 2024-05-08T20:11:42.688 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:42 smithi083 ceph-mon[38695]: pgmap v369: 129 pgs: 129 active+clean; 101 MiB data, 835 MiB used, 714 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.9 MiB/s wr, 176 op/s 2024-05-08T20:11:43.223 INFO:tasks.workunit.client.1.smithi188.stdout: 1 218117 8.96 MB/sec execute 7 sec latency 7.342 ms 2024-05-08T20:11:43.556 INFO:tasks.workunit.client.0.smithi083.stdout: 1 373166 15.66 MB/sec execute 7 sec latency 58.508 ms 2024-05-08T20:11:44.223 INFO:tasks.workunit.client.1.smithi188.stdout: 1 220044 9.39 MB/sec execute 8 sec latency 6.604 ms 2024-05-08T20:11:44.556 INFO:tasks.workunit.client.0.smithi083.stdout: 1 376651 16.08 MB/sec execute 8 sec latency 58.133 ms 2024-05-08T20:11:44.588 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:44 smithi188 ceph-mon[48761]: pgmap v370: 129 pgs: 129 active+clean; 103 MiB data, 848 MiB used, 714 GiB / 715 GiB avail; 8.4 KiB/s rd, 4.1 MiB/s wr, 228 op/s 2024-05-08T20:11:44.631 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:44 smithi083 ceph-mon[38695]: pgmap v370: 129 pgs: 129 active+clean; 103 MiB data, 848 MiB used, 714 GiB / 715 GiB avail; 8.4 KiB/s rd, 4.1 MiB/s wr, 228 op/s 2024-05-08T20:11:45.223 INFO:tasks.workunit.client.1.smithi188.stdout: 1 221577 9.08 MB/sec execute 9 sec latency 14.606 ms 2024-05-08T20:11:45.556 INFO:tasks.workunit.client.0.smithi083.stdout: 1 379208 15.76 MB/sec execute 9 sec latency 47.141 ms 2024-05-08T20:11:46.223 INFO:tasks.workunit.client.1.smithi188.stdout: 1 223489 9.40 MB/sec execute 10 sec latency 6.442 ms 2024-05-08T20:11:46.556 INFO:tasks.workunit.client.0.smithi083.stdout: 1 382042 15.53 MB/sec execute 10 sec latency 54.418 ms 2024-05-08T20:11:46.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:46 smithi188 ceph-mon[48761]: pgmap v371: 129 pgs: 129 active+clean; 104 MiB data, 870 MiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.2 MiB/s wr, 193 op/s 2024-05-08T20:11:46.688 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:46 smithi083 ceph-mon[38695]: pgmap v371: 129 pgs: 129 active+clean; 104 MiB data, 870 MiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.2 MiB/s wr, 193 op/s 2024-05-08T20:11:47.223 INFO:tasks.workunit.client.1.smithi188.stdout: 1 225045 9.17 MB/sec execute 11 sec latency 9.515 ms 2024-05-08T20:11:47.556 INFO:tasks.workunit.client.0.smithi083.stdout: 1 384998 15.65 MB/sec execute 11 sec latency 52.043 ms 2024-05-08T20:11:48.223 INFO:tasks.workunit.client.1.smithi188.stdout: 1 226884 9.18 MB/sec execute 12 sec latency 10.766 ms 2024-05-08T20:11:48.438 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:48 smithi083 ceph-mon[38695]: pgmap v372: 129 pgs: 129 active+clean; 104 MiB data, 876 MiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.9 MiB/s wr, 181 op/s 2024-05-08T20:11:48.556 INFO:tasks.workunit.client.0.smithi083.stdout: 1 387600 15.56 MB/sec execute 12 sec latency 108.398 ms 2024-05-08T20:11:48.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:48 smithi188 ceph-mon[48761]: pgmap v372: 129 pgs: 129 active+clean; 104 MiB data, 876 MiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.9 MiB/s wr, 181 op/s 2024-05-08T20:11:49.223 INFO:tasks.workunit.client.1.smithi188.stdout: 1 228480 9.22 MB/sec execute 13 sec latency 7.661 ms 2024-05-08T20:11:49.556 INFO:tasks.workunit.client.0.smithi083.stdout: 1 391022 15.81 MB/sec execute 13 sec latency 148.070 ms 2024-05-08T20:11:50.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 230369 9.23 MB/sec execute 14 sec latency 10.797 ms 2024-05-08T20:11:50.556 INFO:tasks.workunit.client.0.smithi083.stdout: 1 394504 16.03 MB/sec execute 14 sec latency 54.082 ms 2024-05-08T20:11:50.584 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:50 smithi188 ceph-mon[48761]: pgmap v373: 129 pgs: 129 active+clean; 108 MiB data, 898 MiB used, 714 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.8 MiB/s wr, 248 op/s 2024-05-08T20:11:50.688 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:50 smithi083 ceph-mon[38695]: pgmap v373: 129 pgs: 129 active+clean; 108 MiB data, 898 MiB used, 714 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.8 MiB/s wr, 248 op/s 2024-05-08T20:11:51.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 231815 9.07 MB/sec execute 15 sec latency 12.403 ms 2024-05-08T20:11:51.556 INFO:tasks.workunit.client.0.smithi083.stdout: 1 397995 16.23 MB/sec execute 15 sec latency 54.846 ms 2024-05-08T20:11:52.223 INFO:tasks.workunit.client.1.smithi188.stdout: 1 233874 9.27 MB/sec execute 16 sec latency 5.580 ms 2024-05-08T20:11:52.556 INFO:tasks.workunit.client.0.smithi083.stdout: 1 400636 16.10 MB/sec execute 16 sec latency 58.310 ms 2024-05-08T20:11:52.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:52 smithi188 ceph-mon[48761]: pgmap v374: 129 pgs: 129 active+clean; 108 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 190 op/s 2024-05-08T20:11:52.688 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:52 smithi083 ceph-mon[38695]: pgmap v374: 129 pgs: 129 active+clean; 108 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 190 op/s 2024-05-08T20:11:53.223 INFO:tasks.workunit.client.1.smithi188.stdout: 1 235190 9.12 MB/sec execute 17 sec latency 8.728 ms 2024-05-08T20:11:53.556 INFO:tasks.workunit.client.0.smithi083.stdout: 1 403837 16.17 MB/sec execute 17 sec latency 66.580 ms 2024-05-08T20:11:54.188 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:53 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:11:54.223 INFO:tasks.workunit.client.1.smithi188.stdout: 1 237076 9.20 MB/sec execute 18 sec latency 10.322 ms 2024-05-08T20:11:54.339 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:53 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:11:54.556 INFO:tasks.workunit.client.0.smithi083.stdout: 1 406778 16.03 MB/sec execute 18 sec latency 48.046 ms 2024-05-08T20:11:55.188 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:54 smithi083 ceph-mon[38695]: pgmap v375: 129 pgs: 129 active+clean; 109 MiB data, 907 MiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.0 MiB/s wr, 198 op/s 2024-05-08T20:11:55.223 INFO:tasks.workunit.client.1.smithi188.stdout: 1 238569 9.09 MB/sec execute 19 sec latency 10.048 ms 2024-05-08T20:11:55.339 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:54 smithi188 ceph-mon[48761]: pgmap v375: 129 pgs: 129 active+clean; 109 MiB data, 907 MiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.0 MiB/s wr, 198 op/s 2024-05-08T20:11:55.556 INFO:tasks.workunit.client.0.smithi083.stdout: 1 409839 16.11 MB/sec execute 19 sec latency 58.446 ms 2024-05-08T20:11:56.223 INFO:tasks.workunit.client.1.smithi188.stdout: 1 240440 9.16 MB/sec execute 20 sec latency 6.382 ms 2024-05-08T20:11:56.556 INFO:tasks.workunit.client.0.smithi083.stdout: 1 412979 16.24 MB/sec execute 20 sec latency 45.651 ms 2024-05-08T20:11:56.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:56 smithi188 ceph-mon[48761]: pgmap v376: 129 pgs: 129 active+clean; 112 MiB data, 901 MiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.4 MiB/s wr, 192 op/s 2024-05-08T20:11:56.688 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:56 smithi083 ceph-mon[38695]: pgmap v376: 129 pgs: 129 active+clean; 112 MiB data, 901 MiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.4 MiB/s wr, 192 op/s 2024-05-08T20:11:57.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 241931 9.12 MB/sec execute 21 sec latency 16.274 ms 2024-05-08T20:11:57.556 INFO:tasks.workunit.client.0.smithi083.stdout: 1 415924 16.17 MB/sec execute 21 sec latency 56.067 ms 2024-05-08T20:11:57.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:11:57.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:11:57.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:11:57.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:11:57.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:11:57.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:11:57.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:11:57.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:11:57.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:11:57.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:11:57.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:11:57.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:11:58.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 243707 9.04 MB/sec execute 22 sec latency 11.743 ms 2024-05-08T20:11:58.438 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:11:58 smithi083 ceph-mon[38695]: pgmap v377: 129 pgs: 129 active+clean; 111 MiB data, 903 MiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.0 MiB/s wr, 181 op/s 2024-05-08T20:11:58.556 INFO:tasks.workunit.client.0.smithi083.stdout: 1 419328 16.29 MB/sec execute 22 sec latency 60.550 ms 2024-05-08T20:11:58.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:11:58 smithi188 ceph-mon[48761]: pgmap v377: 129 pgs: 129 active+clean; 111 MiB data, 903 MiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.0 MiB/s wr, 181 op/s 2024-05-08T20:11:59.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 245391 9.15 MB/sec execute 23 sec latency 9.315 ms 2024-05-08T20:11:59.556 INFO:tasks.workunit.client.0.smithi083.stdout: 1 422820 16.40 MB/sec execute 23 sec latency 51.743 ms 2024-05-08T20:12:00.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 247179 9.06 MB/sec execute 24 sec latency 9.119 ms 2024-05-08T20:12:00.556 INFO:tasks.workunit.client.0.smithi083.stdout: 1 426350 16.52 MB/sec execute 24 sec latency 47.759 ms 2024-05-08T20:12:00.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:00 smithi188 ceph-mon[48761]: pgmap v378: 129 pgs: 129 active+clean; 112 MiB data, 889 MiB used, 714 GiB / 715 GiB avail; 9.3 KiB/s rd, 4.8 MiB/s wr, 244 op/s 2024-05-08T20:12:00.688 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:00 smithi083 ceph-mon[38695]: pgmap v378: 129 pgs: 129 active+clean; 112 MiB data, 889 MiB used, 714 GiB / 715 GiB avail; 9.3 KiB/s rd, 4.8 MiB/s wr, 244 op/s 2024-05-08T20:12:01.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 248832 9.18 MB/sec execute 25 sec latency 10.421 ms 2024-05-08T20:12:01.557 INFO:tasks.workunit.client.0.smithi083.stdout: 1 428969 16.42 MB/sec execute 25 sec latency 50.092 ms 2024-05-08T20:12:02.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 250637 9.09 MB/sec execute 26 sec latency 6.320 ms 2024-05-08T20:12:02.557 INFO:tasks.workunit.client.0.smithi083.stdout: 1 431988 16.40 MB/sec execute 26 sec latency 58.453 ms 2024-05-08T20:12:02.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:02 smithi188 ceph-mon[48761]: pgmap v379: 129 pgs: 129 active+clean; 112 MiB data, 892 MiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 184 op/s 2024-05-08T20:12:02.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:02 smithi083 ceph-mon[38695]: pgmap v379: 129 pgs: 129 active+clean; 112 MiB data, 892 MiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 184 op/s 2024-05-08T20:12:03.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 252310 9.21 MB/sec execute 27 sec latency 8.705 ms 2024-05-08T20:12:03.557 INFO:tasks.workunit.client.0.smithi083.stdout: 1 434658 16.29 MB/sec execute 27 sec latency 49.396 ms 2024-05-08T20:12:04.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 254037 9.09 MB/sec execute 28 sec latency 10.876 ms 2024-05-08T20:12:04.557 INFO:tasks.workunit.client.0.smithi083.stdout: 1 437303 16.23 MB/sec execute 28 sec latency 47.040 ms 2024-05-08T20:12:04.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:04 smithi188 ceph-mon[48761]: pgmap v380: 129 pgs: 129 active+clean; 112 MiB data, 900 MiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 3.8 MiB/s wr, 191 op/s 2024-05-08T20:12:04.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:04 smithi083 ceph-mon[38695]: pgmap v380: 129 pgs: 129 active+clean; 112 MiB data, 900 MiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 3.8 MiB/s wr, 191 op/s 2024-05-08T20:12:05.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 255791 9.23 MB/sec execute 29 sec latency 13.325 ms 2024-05-08T20:12:05.557 INFO:tasks.workunit.client.0.smithi083.stdout: 1 440668 16.32 MB/sec execute 29 sec latency 55.246 ms 2024-05-08T20:12:06.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 257433 9.11 MB/sec execute 30 sec latency 6.703 ms 2024-05-08T20:12:06.557 INFO:tasks.workunit.client.0.smithi083.stdout: 1 444144 16.40 MB/sec execute 30 sec latency 55.852 ms 2024-05-08T20:12:06.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:06 smithi188 ceph-mon[48761]: pgmap v381: 129 pgs: 129 active+clean; 115 MiB data, 903 MiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 206 op/s 2024-05-08T20:12:06.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:06 smithi083 ceph-mon[38695]: pgmap v381: 129 pgs: 129 active+clean; 115 MiB data, 903 MiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 206 op/s 2024-05-08T20:12:07.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 259250 9.24 MB/sec execute 31 sec latency 9.745 ms 2024-05-08T20:12:07.557 INFO:tasks.workunit.client.0.smithi083.stdout: 1 447623 16.49 MB/sec execute 31 sec latency 98.958 ms 2024-05-08T20:12:08.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 260815 9.13 MB/sec execute 32 sec latency 4.779 ms 2024-05-08T20:12:08.439 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:08 smithi083 ceph-mon[38695]: pgmap v382: 129 pgs: 129 active+clean; 120 MiB data, 938 MiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 196 op/s 2024-05-08T20:12:08.557 INFO:tasks.workunit.client.0.smithi083.stdout: 1 450141 16.38 MB/sec execute 32 sec latency 49.903 ms 2024-05-08T20:12:08.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:08 smithi188 ceph-mon[48761]: pgmap v382: 129 pgs: 129 active+clean; 120 MiB data, 938 MiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 196 op/s 2024-05-08T20:12:09.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 262645 9.23 MB/sec execute 33 sec latency 10.822 ms 2024-05-08T20:12:09.557 INFO:tasks.workunit.client.0.smithi083.stdout: 1 451234 16.07 MB/sec execute 33 sec latency 298.290 ms 2024-05-08T20:12:09.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:12:09.688 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:12:10.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 264131 9.15 MB/sec execute 34 sec latency 13.596 ms 2024-05-08T20:12:10.439 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:10 smithi083 ceph-mon[38695]: pgmap v383: 129 pgs: 129 active+clean; 120 MiB data, 939 MiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.5 MiB/s wr, 211 op/s 2024-05-08T20:12:10.557 INFO:tasks.workunit.client.0.smithi083.stdout: 1 451307 15.60 MB/sec execute 34 sec latency 516.853 ms 2024-05-08T20:12:10.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:10 smithi188 ceph-mon[48761]: pgmap v383: 129 pgs: 129 active+clean; 120 MiB data, 939 MiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.5 MiB/s wr, 211 op/s 2024-05-08T20:12:11.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 266014 9.24 MB/sec execute 35 sec latency 5.612 ms 2024-05-08T20:12:11.557 INFO:tasks.workunit.client.0.smithi083.stdout: 1 454845 15.70 MB/sec execute 35 sec latency 59.586 ms 2024-05-08T20:12:12.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 267528 9.17 MB/sec execute 36 sec latency 14.058 ms 2024-05-08T20:12:12.557 INFO:tasks.workunit.client.0.smithi083.stdout: 1 459015 16.38 MB/sec execute 36 sec latency 61.550 ms 2024-05-08T20:12:12.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:12 smithi188 ceph-mon[48761]: pgmap v384: 129 pgs: 129 active+clean; 120 MiB data, 942 MiB used, 714 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.7 MiB/s wr, 162 op/s 2024-05-08T20:12:12.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:12 smithi083 ceph-mon[38695]: pgmap v384: 129 pgs: 129 active+clean; 120 MiB data, 942 MiB used, 714 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.7 MiB/s wr, 162 op/s 2024-05-08T20:12:13.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 269382 9.18 MB/sec execute 37 sec latency 10.870 ms 2024-05-08T20:12:13.557 INFO:tasks.workunit.client.0.smithi083.stdout: 1 461649 16.32 MB/sec execute 37 sec latency 60.729 ms 2024-05-08T20:12:14.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 270840 9.11 MB/sec execute 38 sec latency 20.179 ms 2024-05-08T20:12:14.557 INFO:tasks.workunit.client.0.smithi083.stdout: 1 464668 16.31 MB/sec execute 38 sec latency 60.010 ms 2024-05-08T20:12:14.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:14 smithi188 ceph-mon[48761]: pgmap v385: 129 pgs: 129 active+clean; 123 MiB data, 931 MiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.1 MiB/s wr, 202 op/s 2024-05-08T20:12:14.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:14 smithi083 ceph-mon[38695]: pgmap v385: 129 pgs: 129 active+clean; 123 MiB data, 931 MiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.1 MiB/s wr, 202 op/s 2024-05-08T20:12:15.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 272887 9.20 MB/sec execute 39 sec latency 10.484 ms 2024-05-08T20:12:15.557 INFO:tasks.workunit.client.0.smithi083.stdout: 1 467339 16.23 MB/sec execute 39 sec latency 59.731 ms 2024-05-08T20:12:16.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 274273 9.13 MB/sec execute 40 sec latency 14.900 ms 2024-05-08T20:12:16.557 INFO:tasks.workunit.client.0.smithi083.stdout: 1 469982 16.19 MB/sec execute 40 sec latency 58.481 ms 2024-05-08T20:12:16.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:16 smithi188 ceph-mon[48761]: pgmap v386: 129 pgs: 129 active+clean; 122 MiB data, 970 MiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 209 op/s 2024-05-08T20:12:16.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:16 smithi083 ceph-mon[38695]: pgmap v386: 129 pgs: 129 active+clean; 122 MiB data, 970 MiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 209 op/s 2024-05-08T20:12:17.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 276167 9.17 MB/sec execute 41 sec latency 6.409 ms 2024-05-08T20:12:17.557 INFO:tasks.workunit.client.0.smithi083.stdout: 1 473352 16.26 MB/sec execute 41 sec latency 59.228 ms 2024-05-08T20:12:18.225 INFO:tasks.workunit.client.1.smithi188.stdout: 1 277674 9.13 MB/sec execute 42 sec latency 6.453 ms 2024-05-08T20:12:18.439 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:18 smithi083 ceph-mon[38695]: pgmap v387: 129 pgs: 129 active+clean; 122 MiB data, 969 MiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 186 op/s 2024-05-08T20:12:18.557 INFO:tasks.workunit.client.0.smithi083.stdout: 1 476888 16.32 MB/sec execute 42 sec latency 51.256 ms 2024-05-08T20:12:18.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:18 smithi188 ceph-mon[48761]: pgmap v387: 129 pgs: 129 active+clean; 122 MiB data, 969 MiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 186 op/s 2024-05-08T20:12:19.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 279608 9.19 MB/sec execute 43 sec latency 7.007 ms 2024-05-08T20:12:19.558 INFO:tasks.workunit.client.0.smithi083.stdout: 1 480403 16.39 MB/sec execute 43 sec latency 49.658 ms 2024-05-08T20:12:20.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 281116 9.14 MB/sec execute 44 sec latency 10.952 ms 2024-05-08T20:12:20.439 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:20 smithi083 ceph-mon[38695]: pgmap v388: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.6 MiB/s wr, 219 op/s 2024-05-08T20:12:20.557 INFO:tasks.workunit.client.0.smithi083.stdout: 1 483897 16.44 MB/sec execute 44 sec latency 49.879 ms 2024-05-08T20:12:20.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:20 smithi188 ceph-mon[48761]: pgmap v388: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.6 MiB/s wr, 219 op/s 2024-05-08T20:12:21.224 INFO:tasks.workunit.client.1.smithi188.stdout: 1 282861 9.10 MB/sec execute 45 sec latency 3.924 ms 2024-05-08T20:12:21.557 INFO:tasks.workunit.client.0.smithi083.stdout: 1 485636 16.23 MB/sec execute 45 sec latency 115.755 ms 2024-05-08T20:12:22.225 INFO:tasks.workunit.client.1.smithi188.stdout: 1 284565 9.15 MB/sec execute 46 sec latency 9.234 ms 2024-05-08T20:12:22.439 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:22 smithi083 ceph-mon[38695]: pgmap v389: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.0 MiB/s wr, 204 op/s 2024-05-08T20:12:22.557 INFO:tasks.workunit.client.0.smithi083.stdout: 1 488651 16.26 MB/sec execute 46 sec latency 58.207 ms 2024-05-08T20:12:22.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:22 smithi188 ceph-mon[48761]: pgmap v389: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.0 MiB/s wr, 204 op/s 2024-05-08T20:12:23.225 INFO:tasks.workunit.client.1.smithi188.stdout: 1 286345 9.12 MB/sec execute 47 sec latency 5.940 ms 2024-05-08T20:12:23.558 INFO:tasks.workunit.client.0.smithi083.stdout: 1 491301 16.22 MB/sec execute 47 sec latency 55.609 ms 2024-05-08T20:12:24.189 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:12:24.225 INFO:tasks.workunit.client.1.smithi188.stdout: 1 287999 9.17 MB/sec execute 48 sec latency 11.795 ms 2024-05-08T20:12:24.339 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:12:24.558 INFO:tasks.workunit.client.0.smithi083.stdout: 1 494664 16.28 MB/sec execute 48 sec latency 54.815 ms 2024-05-08T20:12:25.144 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:24 smithi083 ceph-mon[38695]: pgmap v390: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.0 MiB/s wr, 198 op/s 2024-05-08T20:12:25.225 INFO:tasks.workunit.client.1.smithi188.stdout: 1 289768 9.12 MB/sec execute 49 sec latency 4.569 ms 2024-05-08T20:12:25.339 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:24 smithi188 ceph-mon[48761]: pgmap v390: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.0 MiB/s wr, 198 op/s 2024-05-08T20:12:25.558 INFO:tasks.workunit.client.0.smithi083.stdout: 1 498200 16.33 MB/sec execute 49 sec latency 58.466 ms 2024-05-08T20:12:26.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 291401 9.18 MB/sec execute 50 sec latency 4.384 ms 2024-05-08T20:12:26.339 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:26 smithi188 ceph-mon[48761]: pgmap v391: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.3 MiB/s wr, 201 op/s 2024-05-08T20:12:26.558 INFO:tasks.workunit.client.0.smithi083.stdout: 1 501679 16.39 MB/sec execute 50 sec latency 47.725 ms 2024-05-08T20:12:26.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:26 smithi083 ceph-mon[38695]: pgmap v391: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.3 MiB/s wr, 201 op/s 2024-05-08T20:12:27.225 INFO:tasks.workunit.client.1.smithi188.stdout: 1 293183 9.13 MB/sec execute 51 sec latency 9.864 ms 2024-05-08T20:12:27.558 INFO:tasks.workunit.client.0.smithi083.stdout: 1 505177 16.44 MB/sec execute 51 sec latency 58.351 ms 2024-05-08T20:12:28.231 INFO:tasks.workunit.client.1.smithi188.stdout: 1 294834 9.19 MB/sec execute 52 sec latency 11.913 ms 2024-05-08T20:12:28.455 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:28 smithi083 ceph-mon[38695]: pgmap v392: 129 pgs: 129 active+clean; 128 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 188 op/s 2024-05-08T20:12:28.558 INFO:tasks.workunit.client.0.smithi083.stdout: 1 508709 16.49 MB/sec execute 52 sec latency 46.322 ms 2024-05-08T20:12:28.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:28 smithi188 ceph-mon[48761]: pgmap v392: 129 pgs: 129 active+clean; 128 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 188 op/s 2024-05-08T20:12:29.225 INFO:tasks.workunit.client.1.smithi188.stdout: 1 296536 9.13 MB/sec execute 53 sec latency 11.691 ms 2024-05-08T20:12:29.558 INFO:tasks.workunit.client.0.smithi083.stdout: 1 511228 16.42 MB/sec execute 53 sec latency 58.513 ms 2024-05-08T20:12:30.225 INFO:tasks.workunit.client.1.smithi188.stdout: 1 298326 9.21 MB/sec execute 54 sec latency 8.383 ms 2024-05-08T20:12:30.558 INFO:tasks.workunit.client.0.smithi083.stdout: 1 514243 16.40 MB/sec execute 54 sec latency 58.456 ms 2024-05-08T20:12:30.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:30 smithi188 ceph-mon[48761]: pgmap v393: 129 pgs: 129 active+clean; 131 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.7 MiB/s wr, 244 op/s 2024-05-08T20:12:30.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:30 smithi083 ceph-mon[38695]: pgmap v393: 129 pgs: 129 active+clean; 131 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.7 MiB/s wr, 244 op/s 2024-05-08T20:12:31.225 INFO:tasks.workunit.client.1.smithi188.stdout: 1 299968 9.14 MB/sec execute 55 sec latency 9.961 ms 2024-05-08T20:12:31.558 INFO:tasks.workunit.client.0.smithi083.stdout: 1 517016 16.38 MB/sec execute 55 sec latency 58.181 ms 2024-05-08T20:12:32.225 INFO:tasks.workunit.client.1.smithi188.stdout: 1 301802 9.21 MB/sec execute 56 sec latency 10.624 ms 2024-05-08T20:12:32.558 INFO:tasks.workunit.client.0.smithi083.stdout: 1 520161 16.42 MB/sec execute 56 sec latency 63.862 ms 2024-05-08T20:12:32.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:32 smithi188 ceph-mon[48761]: pgmap v394: 129 pgs: 129 active+clean; 132 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.8 MiB/s wr, 174 op/s 2024-05-08T20:12:32.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:32 smithi083 ceph-mon[38695]: pgmap v394: 129 pgs: 129 active+clean; 132 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.8 MiB/s wr, 174 op/s 2024-05-08T20:12:33.225 INFO:tasks.workunit.client.1.smithi188.stdout: 1 303387 9.15 MB/sec execute 57 sec latency 11.864 ms 2024-05-08T20:12:33.558 INFO:tasks.workunit.client.0.smithi083.stdout: 1 523097 16.39 MB/sec execute 57 sec latency 48.305 ms 2024-05-08T20:12:34.225 INFO:tasks.workunit.client.1.smithi188.stdout: 1 305265 9.23 MB/sec execute 58 sec latency 6.063 ms 2024-05-08T20:12:34.558 INFO:tasks.workunit.client.0.smithi083.stdout: 1 526572 16.44 MB/sec execute 58 sec latency 46.879 ms 2024-05-08T20:12:34.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:34 smithi188 ceph-mon[48761]: pgmap v395: 129 pgs: 129 active+clean; 131 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.3 KiB/s rd, 3.6 MiB/s wr, 179 op/s 2024-05-08T20:12:34.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:34 smithi083 ceph-mon[38695]: pgmap v395: 129 pgs: 129 active+clean; 131 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.3 KiB/s rd, 3.6 MiB/s wr, 179 op/s 2024-05-08T20:12:35.225 INFO:tasks.workunit.client.1.smithi188.stdout: 1 306829 9.17 MB/sec execute 59 sec latency 7.387 ms 2024-05-08T20:12:35.558 INFO:tasks.workunit.client.0.smithi083.stdout: 1 530048 16.48 MB/sec execute 59 sec latency 47.993 ms 2024-05-08T20:12:36.225 INFO:tasks.workunit.client.1.smithi188.stdout: 1 308696 9.22 MB/sec execute 60 sec latency 10.665 ms 2024-05-08T20:12:36.558 INFO:tasks.workunit.client.0.smithi083.stdout: 1 533576 16.52 MB/sec execute 60 sec latency 48.663 ms 2024-05-08T20:12:36.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:36 smithi188 ceph-mon[48761]: pgmap v396: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.6 MiB/s wr, 228 op/s 2024-05-08T20:12:36.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:36 smithi083 ceph-mon[38695]: pgmap v396: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.6 MiB/s wr, 228 op/s 2024-05-08T20:12:37.225 INFO:tasks.workunit.client.1.smithi188.stdout: 1 310207 9.18 MB/sec execute 61 sec latency 14.087 ms 2024-05-08T20:12:37.558 INFO:tasks.workunit.client.0.smithi083.stdout: 1 537107 16.56 MB/sec execute 61 sec latency 52.464 ms 2024-05-08T20:12:38.225 INFO:tasks.workunit.client.1.smithi188.stdout: 1 312007 9.18 MB/sec execute 62 sec latency 7.709 ms 2024-05-08T20:12:38.439 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:38 smithi083 ceph-mon[38695]: pgmap v397: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.9 MiB/s wr, 180 op/s 2024-05-08T20:12:38.558 INFO:tasks.workunit.client.0.smithi083.stdout: 1 539724 16.53 MB/sec execute 62 sec latency 58.280 ms 2024-05-08T20:12:38.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:38 smithi188 ceph-mon[48761]: pgmap v397: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.9 MiB/s wr, 180 op/s 2024-05-08T20:12:39.225 INFO:tasks.workunit.client.1.smithi188.stdout: 1 313643 9.19 MB/sec execute 63 sec latency 11.183 ms 2024-05-08T20:12:39.558 INFO:tasks.workunit.client.0.smithi083.stdout: 1 542397 16.45 MB/sec execute 63 sec latency 47.456 ms 2024-05-08T20:12:39.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:12:39.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:12:40.225 INFO:tasks.workunit.client.1.smithi188.stdout: 1 315482 9.19 MB/sec execute 64 sec latency 12.651 ms 2024-05-08T20:12:40.559 INFO:tasks.workunit.client.0.smithi083.stdout: 1 545410 16.47 MB/sec execute 64 sec latency 52.558 ms 2024-05-08T20:12:40.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:40 smithi188 ceph-mon[48761]: pgmap v398: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.6 MiB/s wr, 234 op/s 2024-05-08T20:12:40.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:40 smithi083 ceph-mon[38695]: pgmap v398: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.6 MiB/s wr, 234 op/s 2024-05-08T20:12:41.225 INFO:tasks.workunit.client.1.smithi188.stdout: 1 316963 9.15 MB/sec execute 65 sec latency 8.430 ms 2024-05-08T20:12:41.559 INFO:tasks.workunit.client.0.smithi083.stdout: 1 547811 16.43 MB/sec execute 65 sec latency 267.004 ms 2024-05-08T20:12:42.225 INFO:tasks.workunit.client.1.smithi188.stdout: 1 318994 9.20 MB/sec execute 66 sec latency 6.691 ms 2024-05-08T20:12:42.559 INFO:tasks.workunit.client.0.smithi083.stdout: 1 547872 16.18 MB/sec execute 66 sec latency 375.152 ms 2024-05-08T20:12:42.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:42 smithi188 ceph-mon[48761]: pgmap v399: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 185 op/s 2024-05-08T20:12:42.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:42 smithi083 ceph-mon[38695]: pgmap v399: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 185 op/s 2024-05-08T20:12:43.225 INFO:tasks.workunit.client.1.smithi188.stdout: 1 320355 9.16 MB/sec execute 67 sec latency 9.825 ms 2024-05-08T20:12:43.558 INFO:tasks.workunit.client.0.smithi083.stdout: 1 548708 16.01 MB/sec execute 67 sec latency 314.675 ms 2024-05-08T20:12:44.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 322256 9.18 MB/sec execute 68 sec latency 4.354 ms 2024-05-08T20:12:44.559 INFO:tasks.workunit.client.0.smithi083.stdout: 1 551537 15.99 MB/sec execute 68 sec latency 58.395 ms 2024-05-08T20:12:44.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:44 smithi188 ceph-mon[48761]: pgmap v400: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.7 MiB/s wr, 193 op/s 2024-05-08T20:12:44.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:44 smithi083 ceph-mon[38695]: pgmap v400: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.7 MiB/s wr, 193 op/s 2024-05-08T20:12:45.225 INFO:tasks.workunit.client.1.smithi188.stdout: 1 323737 9.16 MB/sec execute 69 sec latency 10.829 ms 2024-05-08T20:12:45.559 INFO:tasks.workunit.client.0.smithi083.stdout: 1 554968 16.03 MB/sec execute 69 sec latency 52.489 ms 2024-05-08T20:12:46.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 325572 9.17 MB/sec execute 70 sec latency 6.391 ms 2024-05-08T20:12:46.559 INFO:tasks.workunit.client.0.smithi083.stdout: 1 558443 16.08 MB/sec execute 70 sec latency 52.424 ms 2024-05-08T20:12:46.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:46 smithi188 ceph-mon[48761]: pgmap v401: 129 pgs: 129 active+clean; 140 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.6 MiB/s wr, 206 op/s 2024-05-08T20:12:46.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:46 smithi083 ceph-mon[38695]: pgmap v401: 129 pgs: 129 active+clean; 140 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.6 MiB/s wr, 206 op/s 2024-05-08T20:12:47.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 327094 9.16 MB/sec execute 71 sec latency 16.400 ms 2024-05-08T20:12:47.559 INFO:tasks.workunit.client.0.smithi083.stdout: 1 561935 16.12 MB/sec execute 71 sec latency 58.319 ms 2024-05-08T20:12:48.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 328826 9.14 MB/sec execute 72 sec latency 9.624 ms 2024-05-08T20:12:48.439 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:48 smithi083 ceph-mon[38695]: pgmap v402: 129 pgs: 129 active+clean; 143 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.9 MiB/s wr, 189 op/s 2024-05-08T20:12:48.559 INFO:tasks.workunit.client.0.smithi083.stdout: 1 564448 16.08 MB/sec execute 72 sec latency 83.049 ms 2024-05-08T20:12:48.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:48 smithi188 ceph-mon[48761]: pgmap v402: 129 pgs: 129 active+clean; 143 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.9 MiB/s wr, 189 op/s 2024-05-08T20:12:49.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 330434 9.17 MB/sec execute 73 sec latency 14.166 ms 2024-05-08T20:12:49.559 INFO:tasks.workunit.client.0.smithi083.stdout: 1 567273 16.04 MB/sec execute 73 sec latency 57.784 ms 2024-05-08T20:12:50.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 332208 9.14 MB/sec execute 74 sec latency 4.847 ms 2024-05-08T20:12:50.468 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:50 smithi083 ceph-mon[38695]: pgmap v403: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.3 KiB/s rd, 4.5 MiB/s wr, 207 op/s 2024-05-08T20:12:50.559 INFO:tasks.workunit.client.0.smithi083.stdout: 1 569984 16.05 MB/sec execute 74 sec latency 58.449 ms 2024-05-08T20:12:50.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:50 smithi188 ceph-mon[48761]: pgmap v403: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.3 KiB/s rd, 4.5 MiB/s wr, 207 op/s 2024-05-08T20:12:51.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 333840 9.18 MB/sec execute 75 sec latency 11.480 ms 2024-05-08T20:12:51.559 INFO:tasks.workunit.client.0.smithi083.stdout: 1 572811 16.03 MB/sec execute 75 sec latency 67.055 ms 2024-05-08T20:12:52.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 335464 9.13 MB/sec execute 76 sec latency 7.467 ms 2024-05-08T20:12:52.559 INFO:tasks.workunit.client.0.smithi083.stdout: 1 576241 16.07 MB/sec execute 76 sec latency 58.958 ms 2024-05-08T20:12:52.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:52 smithi188 ceph-mon[48761]: pgmap v404: 129 pgs: 129 active+clean; 143 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.8 MiB/s wr, 163 op/s 2024-05-08T20:12:52.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:52 smithi083 ceph-mon[38695]: pgmap v404: 129 pgs: 129 active+clean; 143 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.8 MiB/s wr, 163 op/s 2024-05-08T20:12:53.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 337232 9.18 MB/sec execute 77 sec latency 14.010 ms 2024-05-08T20:12:53.559 INFO:tasks.workunit.client.0.smithi083.stdout: 1 579663 16.10 MB/sec execute 77 sec latency 101.902 ms 2024-05-08T20:12:54.190 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:53 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:12:54.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 338754 9.14 MB/sec execute 78 sec latency 10.372 ms 2024-05-08T20:12:54.340 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:53 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:12:54.560 INFO:tasks.workunit.client.0.smithi083.stdout: 1 582271 16.08 MB/sec execute 78 sec latency 86.876 ms 2024-05-08T20:12:55.189 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:54 smithi083 ceph-mon[38695]: pgmap v405: 129 pgs: 129 active+clean; 145 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 3.8 MiB/s wr, 201 op/s 2024-05-08T20:12:55.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 340557 9.18 MB/sec execute 79 sec latency 6.589 ms 2024-05-08T20:12:55.339 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:54 smithi188 ceph-mon[48761]: pgmap v405: 129 pgs: 129 active+clean; 145 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 3.8 MiB/s wr, 201 op/s 2024-05-08T20:12:55.559 INFO:tasks.workunit.client.0.smithi083.stdout: 1 585009 16.03 MB/sec execute 79 sec latency 59.889 ms 2024-05-08T20:12:56.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 342051 9.15 MB/sec execute 80 sec latency 10.766 ms 2024-05-08T20:12:56.559 INFO:tasks.workunit.client.0.smithi083.stdout: 1 587963 16.04 MB/sec execute 80 sec latency 58.437 ms 2024-05-08T20:12:56.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:56 smithi188 ceph-mon[48761]: pgmap v406: 129 pgs: 129 active+clean; 146 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 205 op/s 2024-05-08T20:12:56.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:56 smithi083 ceph-mon[38695]: pgmap v406: 129 pgs: 129 active+clean; 146 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 205 op/s 2024-05-08T20:12:57.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 343853 9.15 MB/sec execute 81 sec latency 8.327 ms 2024-05-08T20:12:57.559 INFO:tasks.workunit.client.0.smithi083.stdout: 1 590547 16.02 MB/sec execute 81 sec latency 51.156 ms 2024-05-08T20:12:57.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:12:57.839 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:12:58.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 345254 9.12 MB/sec execute 82 sec latency 15.832 ms 2024-05-08T20:12:58.559 INFO:tasks.workunit.client.0.smithi083.stdout: 1 593972 16.06 MB/sec execute 82 sec latency 58.524 ms 2024-05-08T20:12:58.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:58 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:12:58.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:58 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:12:58.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:58 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:12:58.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:12:58 smithi083 ceph-mon[38695]: pgmap v407: 129 pgs: 129 active+clean; 145 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.7 MiB/s wr, 185 op/s 2024-05-08T20:12:58.840 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:58 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:12:59.064 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:58 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:12:59.065 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:58 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:12:59.065 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:12:58 smithi188 ceph-mon[48761]: pgmap v407: 129 pgs: 129 active+clean; 145 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.7 MiB/s wr, 185 op/s 2024-05-08T20:12:59.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 347276 9.15 MB/sec execute 83 sec latency 7.249 ms 2024-05-08T20:12:59.559 INFO:tasks.workunit.client.0.smithi083.stdout: 1 597449 16.09 MB/sec execute 83 sec latency 75.111 ms 2024-05-08T20:13:00.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 348614 9.12 MB/sec execute 84 sec latency 13.467 ms 2024-05-08T20:13:00.559 INFO:tasks.workunit.client.0.smithi083.stdout: 1 600938 16.13 MB/sec execute 84 sec latency 58.476 ms 2024-05-08T20:13:00.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:00 smithi188 ceph-mon[48761]: pgmap v408: 129 pgs: 129 active+clean; 150 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.6 MiB/s wr, 219 op/s 2024-05-08T20:13:00.597 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:00 smithi083 ceph-mon[38695]: pgmap v408: 129 pgs: 129 active+clean; 150 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.6 MiB/s wr, 219 op/s 2024-05-08T20:13:01.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 350505 9.14 MB/sec execute 85 sec latency 4.769 ms 2024-05-08T20:13:01.560 INFO:tasks.workunit.client.0.smithi083.stdout: 1 603556 16.10 MB/sec execute 85 sec latency 58.244 ms 2024-05-08T20:13:02.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 351981 9.12 MB/sec execute 86 sec latency 7.921 ms 2024-05-08T20:13:02.560 INFO:tasks.workunit.client.0.smithi083.stdout: 1 606283 16.06 MB/sec execute 86 sec latency 49.786 ms 2024-05-08T20:13:02.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:02 smithi188 ceph-mon[48761]: pgmap v409: 129 pgs: 129 active+clean; 151 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.0 MiB/s wr, 194 op/s 2024-05-08T20:13:02.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:02 smithi083 ceph-mon[38695]: pgmap v409: 129 pgs: 129 active+clean; 151 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.0 MiB/s wr, 194 op/s 2024-05-08T20:13:03.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 353749 9.10 MB/sec execute 87 sec latency 8.337 ms 2024-05-08T20:13:03.560 INFO:tasks.workunit.client.0.smithi083.stdout: 1 609234 16.07 MB/sec execute 87 sec latency 66.782 ms 2024-05-08T20:13:04.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 355395 9.13 MB/sec execute 88 sec latency 9.339 ms 2024-05-08T20:13:04.560 INFO:tasks.workunit.client.0.smithi083.stdout: 1 611821 16.05 MB/sec execute 88 sec latency 66.757 ms 2024-05-08T20:13:04.589 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:04 smithi188 ceph-mon[48761]: pgmap v410: 129 pgs: 129 active+clean; 151 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.9 MiB/s wr, 190 op/s 2024-05-08T20:13:04.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:04 smithi083 ceph-mon[38695]: pgmap v410: 129 pgs: 129 active+clean; 151 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.9 MiB/s wr, 190 op/s 2024-05-08T20:13:05.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 357172 9.11 MB/sec execute 89 sec latency 8.992 ms 2024-05-08T20:13:05.560 INFO:tasks.workunit.client.0.smithi083.stdout: 1 615244 16.08 MB/sec execute 89 sec latency 49.898 ms 2024-05-08T20:13:06.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 358835 9.14 MB/sec execute 90 sec latency 9.511 ms 2024-05-08T20:13:06.468 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:06 smithi188 ceph-mon[48761]: pgmap v411: 129 pgs: 129 active+clean; 152 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.3 MiB/s wr, 197 op/s 2024-05-08T20:13:06.560 INFO:tasks.workunit.client.0.smithi083.stdout: 1 618721 16.12 MB/sec execute 90 sec latency 50.322 ms 2024-05-08T20:13:06.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:06 smithi083 ceph-mon[38695]: pgmap v411: 129 pgs: 129 active+clean; 152 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.3 MiB/s wr, 197 op/s 2024-05-08T20:13:07.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 360618 9.11 MB/sec execute 91 sec latency 7.866 ms 2024-05-08T20:13:07.560 INFO:tasks.workunit.client.0.smithi083.stdout: 1 622209 16.15 MB/sec execute 91 sec latency 55.896 ms 2024-05-08T20:13:08.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 362275 9.15 MB/sec execute 92 sec latency 10.867 ms 2024-05-08T20:13:08.440 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:08 smithi083 ceph-mon[38695]: pgmap v412: 129 pgs: 129 active+clean; 151 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.8 MiB/s wr, 186 op/s 2024-05-08T20:13:08.560 INFO:tasks.workunit.client.0.smithi083.stdout: 1 624820 16.13 MB/sec execute 92 sec latency 58.395 ms 2024-05-08T20:13:08.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:08 smithi188 ceph-mon[48761]: pgmap v412: 129 pgs: 129 active+clean; 151 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.8 MiB/s wr, 186 op/s 2024-05-08T20:13:09.227 INFO:tasks.workunit.client.1.smithi188.stdout: 1 364066 9.11 MB/sec execute 93 sec latency 5.499 ms 2024-05-08T20:13:09.560 INFO:tasks.workunit.client.0.smithi083.stdout: 1 627494 16.08 MB/sec execute 93 sec latency 58.429 ms 2024-05-08T20:13:09.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:13:09.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:13:10.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 365747 9.15 MB/sec execute 94 sec latency 9.082 ms 2024-05-08T20:13:10.560 INFO:tasks.workunit.client.0.smithi083.stdout: 1 630257 16.09 MB/sec execute 94 sec latency 58.061 ms 2024-05-08T20:13:10.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:10 smithi188 ceph-mon[48761]: pgmap v413: 129 pgs: 129 active+clean; 158 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.0 MiB/s wr, 254 op/s 2024-05-08T20:13:10.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:10 smithi083 ceph-mon[38695]: pgmap v413: 129 pgs: 129 active+clean; 158 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.0 MiB/s wr, 254 op/s 2024-05-08T20:13:11.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 367422 9.12 MB/sec execute 95 sec latency 12.228 ms 2024-05-08T20:13:11.560 INFO:tasks.workunit.client.0.smithi083.stdout: 1 633093 16.08 MB/sec execute 95 sec latency 52.869 ms 2024-05-08T20:13:12.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 369186 9.16 MB/sec execute 96 sec latency 16.887 ms 2024-05-08T20:13:12.560 INFO:tasks.workunit.client.0.smithi083.stdout: 1 636517 16.11 MB/sec execute 96 sec latency 58.070 ms 2024-05-08T20:13:12.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:12 smithi188 ceph-mon[48761]: pgmap v414: 129 pgs: 129 active+clean; 158 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.7 MiB/s wr, 175 op/s 2024-05-08T20:13:12.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:12 smithi083 ceph-mon[38695]: pgmap v414: 129 pgs: 129 active+clean; 158 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.7 MiB/s wr, 175 op/s 2024-05-08T20:13:13.226 INFO:tasks.workunit.client.1.smithi188.stdout: 1 370757 9.12 MB/sec execute 97 sec latency 11.692 ms 2024-05-08T20:13:13.560 INFO:tasks.workunit.client.0.smithi083.stdout: 1 639996 16.13 MB/sec execute 97 sec latency 49.774 ms 2024-05-08T20:13:14.227 INFO:tasks.workunit.client.1.smithi188.stdout: 1 372630 9.17 MB/sec execute 98 sec latency 9.400 ms 2024-05-08T20:13:14.560 INFO:tasks.workunit.client.0.smithi083.stdout: 1 640894 16.02 MB/sec execute 98 sec latency 280.615 ms 2024-05-08T20:13:14.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:14 smithi188 ceph-mon[48761]: pgmap v415: 129 pgs: 129 active+clean; 157 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.3 KiB/s rd, 3.7 MiB/s wr, 188 op/s 2024-05-08T20:13:14.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:14 smithi083 ceph-mon[38695]: pgmap v415: 129 pgs: 129 active+clean; 157 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.3 KiB/s rd, 3.7 MiB/s wr, 188 op/s 2024-05-08T20:13:15.227 INFO:tasks.workunit.client.1.smithi188.stdout: 1 374121 9.13 MB/sec execute 99 sec latency 15.914 ms 2024-05-08T20:13:15.560 INFO:tasks.workunit.client.0.smithi083.stdout: 1 641735 15.88 MB/sec execute 99 sec latency 449.422 ms 2024-05-08T20:13:16.227 INFO:tasks.workunit.client.1.smithi188.stdout: 1 375985 9.16 MB/sec execute 100 sec latency 7.316 ms 2024-05-08T20:13:16.560 INFO:tasks.workunit.client.0.smithi083.stdout: 1 644440 15.89 MB/sec execute 100 sec latency 59.253 ms 2024-05-08T20:13:16.586 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:16 smithi188 ceph-mon[48761]: pgmap v416: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 230 op/s 2024-05-08T20:13:16.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:16 smithi083 ceph-mon[38695]: pgmap v416: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 230 op/s 2024-05-08T20:13:17.227 INFO:tasks.workunit.client.1.smithi188.stdout: 1 377504 9.14 MB/sec execute 101 sec latency 9.551 ms 2024-05-08T20:13:17.560 INFO:tasks.workunit.client.0.smithi083.stdout: 1 647236 15.88 MB/sec execute 101 sec latency 66.763 ms 2024-05-08T20:13:18.227 INFO:tasks.workunit.client.1.smithi188.stdout: 1 379347 9.14 MB/sec execute 102 sec latency 9.688 ms 2024-05-08T20:13:18.439 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:18 smithi083 ceph-mon[38695]: pgmap v417: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 179 op/s 2024-05-08T20:13:18.560 INFO:tasks.workunit.client.0.smithi083.stdout: 1 650693 15.91 MB/sec execute 102 sec latency 48.951 ms 2024-05-08T20:13:18.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:18 smithi188 ceph-mon[48761]: pgmap v417: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 179 op/s 2024-05-08T20:13:19.227 INFO:tasks.workunit.client.1.smithi188.stdout: 1 380951 9.15 MB/sec execute 103 sec latency 15.045 ms 2024-05-08T20:13:19.560 INFO:tasks.workunit.client.0.smithi083.stdout: 1 654178 15.94 MB/sec execute 103 sec latency 58.601 ms 2024-05-08T20:13:20.227 INFO:tasks.workunit.client.1.smithi188.stdout: 1 382866 9.15 MB/sec execute 104 sec latency 12.395 ms 2024-05-08T20:13:20.560 INFO:tasks.workunit.client.0.smithi083.stdout: 1 657664 15.97 MB/sec execute 104 sec latency 58.349 ms 2024-05-08T20:13:20.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:20 smithi188 ceph-mon[48761]: pgmap v418: 129 pgs: 129 active+clean; 162 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.8 MiB/s wr, 238 op/s 2024-05-08T20:13:20.689 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:20 smithi083 ceph-mon[38695]: pgmap v418: 129 pgs: 129 active+clean; 162 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.8 MiB/s wr, 238 op/s 2024-05-08T20:13:21.227 INFO:tasks.workunit.client.1.smithi188.stdout: 1 384268 9.13 MB/sec execute 105 sec latency 9.302 ms 2024-05-08T20:13:21.560 INFO:tasks.workunit.client.0.smithi083.stdout: 1 660177 15.94 MB/sec execute 105 sec latency 59.659 ms 2024-05-08T20:13:22.227 INFO:tasks.workunit.client.1.smithi188.stdout: 1 386302 9.14 MB/sec execute 106 sec latency 12.856 ms 2024-05-08T20:13:22.561 INFO:tasks.workunit.client.0.smithi083.stdout: 1 663451 15.96 MB/sec execute 106 sec latency 58.422 ms 2024-05-08T20:13:22.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:22 smithi188 ceph-mon[48761]: pgmap v419: 129 pgs: 129 active+clean; 161 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.6 MiB/s wr, 165 op/s 2024-05-08T20:13:22.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:22 smithi083 ceph-mon[38695]: pgmap v419: 129 pgs: 129 active+clean; 161 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.6 MiB/s wr, 165 op/s 2024-05-08T20:13:23.227 INFO:tasks.workunit.client.1.smithi188.stdout: 1 387674 9.12 MB/sec execute 107 sec latency 6.086 ms 2024-05-08T20:13:23.561 INFO:tasks.workunit.client.0.smithi083.stdout: 1 665967 15.93 MB/sec execute 107 sec latency 58.422 ms 2024-05-08T20:13:24.190 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:13:24.227 INFO:tasks.workunit.client.1.smithi188.stdout: 1 389586 9.14 MB/sec execute 108 sec latency 5.597 ms 2024-05-08T20:13:24.340 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:13:24.561 INFO:tasks.workunit.client.0.smithi083.stdout: 1 668615 15.91 MB/sec execute 108 sec latency 58.450 ms 2024-05-08T20:13:25.189 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:24 smithi083 ceph-mon[38695]: pgmap v420: 129 pgs: 129 active+clean; 162 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 3.8 MiB/s wr, 192 op/s 2024-05-08T20:13:25.228 INFO:tasks.workunit.client.1.smithi188.stdout: 1 391093 9.13 MB/sec execute 109 sec latency 20.359 ms 2024-05-08T20:13:25.340 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:24 smithi188 ceph-mon[48761]: pgmap v420: 129 pgs: 129 active+clean; 162 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 3.8 MiB/s wr, 192 op/s 2024-05-08T20:13:25.561 INFO:tasks.workunit.client.0.smithi083.stdout: 1 671975 15.94 MB/sec execute 109 sec latency 58.276 ms 2024-05-08T20:13:26.227 INFO:tasks.workunit.client.1.smithi188.stdout: 1 392837 9.11 MB/sec execute 110 sec latency 9.915 ms 2024-05-08T20:13:26.561 INFO:tasks.workunit.client.0.smithi083.stdout: 1 675504 15.97 MB/sec execute 110 sec latency 86.977 ms 2024-05-08T20:13:26.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:26 smithi188 ceph-mon[48761]: pgmap v421: 129 pgs: 129 active+clean; 164 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.5 MiB/s wr, 211 op/s 2024-05-08T20:13:26.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:26 smithi083 ceph-mon[38695]: pgmap v421: 129 pgs: 129 active+clean; 164 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.5 MiB/s wr, 211 op/s 2024-05-08T20:13:27.227 INFO:tasks.workunit.client.1.smithi188.stdout: 1 394501 9.13 MB/sec execute 111 sec latency 7.659 ms 2024-05-08T20:13:27.561 INFO:tasks.workunit.client.0.smithi083.stdout: 1 678991 15.99 MB/sec execute 111 sec latency 66.731 ms 2024-05-08T20:13:28.227 INFO:tasks.workunit.client.1.smithi188.stdout: 1 396235 9.12 MB/sec execute 112 sec latency 8.476 ms 2024-05-08T20:13:28.440 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:28 smithi083 ceph-mon[38695]: pgmap v422: 129 pgs: 129 active+clean; 163 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.7 MiB/s wr, 160 op/s 2024-05-08T20:13:28.561 INFO:tasks.workunit.client.0.smithi083.stdout: 1 681503 15.97 MB/sec execute 112 sec latency 60.964 ms 2024-05-08T20:13:28.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:28 smithi188 ceph-mon[48761]: pgmap v422: 129 pgs: 129 active+clean; 163 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.7 MiB/s wr, 160 op/s 2024-05-08T20:13:29.228 INFO:tasks.workunit.client.1.smithi188.stdout: 1 397907 9.14 MB/sec execute 113 sec latency 9.424 ms 2024-05-08T20:13:29.561 INFO:tasks.workunit.client.0.smithi083.stdout: 1 684336 15.95 MB/sec execute 113 sec latency 52.079 ms 2024-05-08T20:13:30.227 INFO:tasks.workunit.client.1.smithi188.stdout: 1 399695 9.12 MB/sec execute 114 sec latency 10.491 ms 2024-05-08T20:13:30.561 INFO:tasks.workunit.client.0.smithi083.stdout: 1 687294 15.95 MB/sec execute 114 sec latency 58.479 ms 2024-05-08T20:13:30.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:30 smithi188 ceph-mon[48761]: pgmap v423: 129 pgs: 129 active+clean; 166 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.7 MiB/s wr, 238 op/s 2024-05-08T20:13:30.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:30 smithi083 ceph-mon[38695]: pgmap v423: 129 pgs: 129 active+clean; 166 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.7 MiB/s wr, 238 op/s 2024-05-08T20:13:31.228 INFO:tasks.workunit.client.1.smithi188.stdout: 1 401345 9.15 MB/sec execute 115 sec latency 11.168 ms 2024-05-08T20:13:31.561 INFO:tasks.workunit.client.0.smithi083.stdout: 1 689938 15.94 MB/sec execute 115 sec latency 51.018 ms 2024-05-08T20:13:32.228 INFO:tasks.workunit.client.1.smithi188.stdout: 1 403120 9.12 MB/sec execute 116 sec latency 17.716 ms 2024-05-08T20:13:32.561 INFO:tasks.workunit.client.0.smithi083.stdout: 1 693299 15.97 MB/sec execute 116 sec latency 59.031 ms 2024-05-08T20:13:32.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:32 smithi188 ceph-mon[48761]: pgmap v424: 129 pgs: 129 active+clean; 167 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.8 MiB/s wr, 172 op/s 2024-05-08T20:13:32.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:32 smithi083 ceph-mon[38695]: pgmap v424: 129 pgs: 129 active+clean; 167 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.8 MiB/s wr, 172 op/s 2024-05-08T20:13:33.228 INFO:tasks.workunit.client.1.smithi188.stdout: 1 404827 9.15 MB/sec execute 117 sec latency 6.392 ms 2024-05-08T20:13:33.561 INFO:tasks.workunit.client.0.smithi083.stdout: 1 696778 15.99 MB/sec execute 117 sec latency 49.905 ms 2024-05-08T20:13:34.228 INFO:tasks.workunit.client.1.smithi188.stdout: 1 406505 9.12 MB/sec execute 118 sec latency 8.285 ms 2024-05-08T20:13:34.561 INFO:tasks.workunit.client.0.smithi083.stdout: 1 700264 16.02 MB/sec execute 118 sec latency 58.507 ms 2024-05-08T20:13:34.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:34 smithi188 ceph-mon[48761]: pgmap v425: 129 pgs: 129 active+clean; 168 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 3.9 MiB/s wr, 183 op/s 2024-05-08T20:13:34.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:34 smithi083 ceph-mon[38695]: pgmap v425: 129 pgs: 129 active+clean; 168 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 3.9 MiB/s wr, 183 op/s 2024-05-08T20:13:35.228 INFO:tasks.workunit.client.1.smithi188.stdout: 1 408287 9.16 MB/sec execute 119 sec latency 17.436 ms 2024-05-08T20:13:35.561 INFO:tasks.workunit.client.0.smithi083.stdout: 1 702877 16.00 MB/sec execute 119 sec latency 60.148 ms 2024-05-08T20:13:36.228 INFO:tasks.workunit.client.1.smithi188.stdout: 1 409875 9.13 MB/sec execute 120 sec latency 22.175 ms 2024-05-08T20:13:36.440 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:36 smithi083 ceph-mon[38695]: pgmap v426: 129 pgs: 129 active+clean; 172 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.6 MiB/s wr, 214 op/s 2024-05-08T20:13:36.561 INFO:tasks.workunit.client.0.smithi083.stdout: 1 705548 15.97 MB/sec execute 120 sec latency 59.775 ms 2024-05-08T20:13:36.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:36 smithi188 ceph-mon[48761]: pgmap v426: 129 pgs: 129 active+clean; 172 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.6 MiB/s wr, 214 op/s 2024-05-08T20:13:37.228 INFO:tasks.workunit.client.1.smithi188.stdout: 1 411714 9.16 MB/sec execute 121 sec latency 11.075 ms 2024-05-08T20:13:37.561 INFO:tasks.workunit.client.0.smithi083.stdout: 1 708563 15.98 MB/sec execute 121 sec latency 58.239 ms 2024-05-08T20:13:38.228 INFO:tasks.workunit.client.1.smithi188.stdout: 1 413184 9.13 MB/sec execute 122 sec latency 14.771 ms 2024-05-08T20:13:38.440 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:38 smithi083 ceph-mon[38695]: pgmap v427: 129 pgs: 129 active+clean; 172 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 188 op/s 2024-05-08T20:13:38.561 INFO:tasks.workunit.client.0.smithi083.stdout: 1 711213 15.97 MB/sec execute 122 sec latency 61.077 ms 2024-05-08T20:13:38.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:38 smithi188 ceph-mon[48761]: pgmap v427: 129 pgs: 129 active+clean; 172 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 188 op/s 2024-05-08T20:13:39.228 INFO:tasks.workunit.client.1.smithi188.stdout: 1 415044 9.16 MB/sec execute 123 sec latency 10.135 ms 2024-05-08T20:13:39.561 INFO:tasks.workunit.client.0.smithi083.stdout: 1 714648 15.99 MB/sec execute 123 sec latency 61.804 ms 2024-05-08T20:13:39.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:13:39.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:13:40.228 INFO:tasks.workunit.client.1.smithi188.stdout: 1 416532 9.14 MB/sec execute 124 sec latency 6.423 ms 2024-05-08T20:13:40.561 INFO:tasks.workunit.client.0.smithi083.stdout: 1 718126 16.02 MB/sec execute 124 sec latency 54.626 ms 2024-05-08T20:13:40.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:40 smithi188 ceph-mon[48761]: pgmap v428: 129 pgs: 129 active+clean; 176 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.3 KiB/s rd, 4.9 MiB/s wr, 250 op/s 2024-05-08T20:13:40.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:40 smithi083 ceph-mon[38695]: pgmap v428: 129 pgs: 129 active+clean; 176 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.3 KiB/s rd, 4.9 MiB/s wr, 250 op/s 2024-05-08T20:13:41.228 INFO:tasks.workunit.client.1.smithi188.stdout: 1 418361 9.14 MB/sec execute 125 sec latency 14.332 ms 2024-05-08T20:13:41.561 INFO:tasks.workunit.client.0.smithi083.stdout: 1 721669 16.04 MB/sec execute 125 sec latency 48.894 ms 2024-05-08T20:13:42.228 INFO:tasks.workunit.client.1.smithi188.stdout: 1 419801 9.12 MB/sec execute 126 sec latency 10.338 ms 2024-05-08T20:13:42.561 INFO:tasks.workunit.client.0.smithi083.stdout: 1 725212 16.07 MB/sec execute 126 sec latency 66.413 ms 2024-05-08T20:13:42.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:42 smithi188 ceph-mon[48761]: pgmap v429: 129 pgs: 129 active+clean; 176 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.7 MiB/s wr, 169 op/s 2024-05-08T20:13:42.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:42 smithi083 ceph-mon[38695]: pgmap v429: 129 pgs: 129 active+clean; 176 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.7 MiB/s wr, 169 op/s 2024-05-08T20:13:43.228 INFO:tasks.workunit.client.1.smithi188.stdout: 1 421851 9.15 MB/sec execute 127 sec latency 7.489 ms 2024-05-08T20:13:43.561 INFO:tasks.workunit.client.0.smithi083.stdout: 1 728691 16.09 MB/sec execute 127 sec latency 50.172 ms 2024-05-08T20:13:44.228 INFO:tasks.workunit.client.1.smithi188.stdout: 1 423187 9.13 MB/sec execute 128 sec latency 4.841 ms 2024-05-08T20:13:44.561 INFO:tasks.workunit.client.0.smithi083.stdout: 1 732233 16.11 MB/sec execute 128 sec latency 59.464 ms 2024-05-08T20:13:44.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:44 smithi188 ceph-mon[48761]: pgmap v430: 129 pgs: 129 active+clean; 175 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.7 MiB/s wr, 180 op/s 2024-05-08T20:13:44.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:44 smithi083 ceph-mon[38695]: pgmap v430: 129 pgs: 129 active+clean; 175 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.7 MiB/s wr, 180 op/s 2024-05-08T20:13:45.229 INFO:tasks.workunit.client.1.smithi188.stdout: 1 425032 9.13 MB/sec execute 129 sec latency 12.924 ms 2024-05-08T20:13:45.562 INFO:tasks.workunit.client.0.smithi083.stdout: 1 735772 16.13 MB/sec execute 129 sec latency 56.148 ms 2024-05-08T20:13:46.229 INFO:tasks.workunit.client.1.smithi188.stdout: 1 426537 9.12 MB/sec execute 130 sec latency 9.417 ms 2024-05-08T20:13:46.440 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:46 smithi083 ceph-mon[38695]: pgmap v431: 129 pgs: 129 active+clean; 175 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.4 MiB/s wr, 204 op/s 2024-05-08T20:13:46.562 INFO:tasks.workunit.client.0.smithi083.stdout: 1 737957 16.10 MB/sec execute 130 sec latency 283.954 ms 2024-05-08T20:13:46.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:46 smithi188 ceph-mon[48761]: pgmap v431: 129 pgs: 129 active+clean; 175 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.4 MiB/s wr, 204 op/s 2024-05-08T20:13:47.229 INFO:tasks.workunit.client.1.smithi188.stdout: 1 428233 9.11 MB/sec execute 131 sec latency 22.060 ms 2024-05-08T20:13:47.562 INFO:tasks.workunit.client.0.smithi083.stdout: 1 738035 15.98 MB/sec execute 131 sec latency 557.998 ms 2024-05-08T20:13:48.229 INFO:tasks.workunit.client.1.smithi188.stdout: 1 429836 9.13 MB/sec execute 132 sec latency 11.060 ms 2024-05-08T20:13:48.440 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:48 smithi083 ceph-mon[38695]: pgmap v432: 129 pgs: 129 active+clean; 179 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 191 op/s 2024-05-08T20:13:48.562 INFO:tasks.workunit.client.0.smithi083.stdout: 1 739576 15.92 MB/sec execute 132 sec latency 341.825 ms 2024-05-08T20:13:48.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:48 smithi188 ceph-mon[48761]: pgmap v432: 129 pgs: 129 active+clean; 179 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 191 op/s 2024-05-08T20:13:49.229 INFO:tasks.workunit.client.1.smithi188.stdout: 1 431590 9.11 MB/sec execute 133 sec latency 21.114 ms 2024-05-08T20:13:49.562 INFO:tasks.workunit.client.0.smithi083.stdout: 1 742936 15.94 MB/sec execute 133 sec latency 57.471 ms 2024-05-08T20:13:50.230 INFO:tasks.workunit.client.1.smithi188.stdout: 1 433246 9.13 MB/sec execute 134 sec latency 5.227 ms 2024-05-08T20:13:50.562 INFO:tasks.workunit.client.0.smithi083.stdout: 1 746416 15.96 MB/sec execute 134 sec latency 57.291 ms 2024-05-08T20:13:50.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:50 smithi188 ceph-mon[48761]: pgmap v433: 129 pgs: 129 active+clean; 178 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.5 MiB/s wr, 223 op/s 2024-05-08T20:13:50.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:50 smithi083 ceph-mon[38695]: pgmap v433: 129 pgs: 129 active+clean; 178 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.5 MiB/s wr, 223 op/s 2024-05-08T20:13:51.230 INFO:tasks.workunit.client.1.smithi188.stdout: 1 434989 9.11 MB/sec execute 135 sec latency 9.756 ms 2024-05-08T20:13:51.562 INFO:tasks.workunit.client.0.smithi083.stdout: 1 749901 15.98 MB/sec execute 135 sec latency 67.478 ms 2024-05-08T20:13:52.231 INFO:tasks.workunit.client.1.smithi188.stdout: 1 436705 9.14 MB/sec execute 136 sec latency 12.174 ms 2024-05-08T20:13:52.340 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:52 smithi188 ceph-mon[48761]: pgmap v434: 129 pgs: 129 active+clean; 179 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.5 MiB/s wr, 169 op/s 2024-05-08T20:13:52.562 INFO:tasks.workunit.client.0.smithi083.stdout: 1 752513 15.97 MB/sec execute 136 sec latency 80.532 ms 2024-05-08T20:13:52.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:52 smithi083 ceph-mon[38695]: pgmap v434: 129 pgs: 129 active+clean; 179 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.5 MiB/s wr, 169 op/s 2024-05-08T20:13:53.231 INFO:tasks.workunit.client.1.smithi188.stdout: 1 438352 9.11 MB/sec execute 137 sec latency 11.234 ms 2024-05-08T20:13:53.562 INFO:tasks.workunit.client.0.smithi083.stdout: 1 755123 15.94 MB/sec execute 137 sec latency 74.710 ms 2024-05-08T20:13:54.190 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:53 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:13:54.231 INFO:tasks.workunit.client.1.smithi188.stdout: 1 440155 9.14 MB/sec execute 138 sec latency 9.025 ms 2024-05-08T20:13:54.340 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:53 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:13:54.562 INFO:tasks.workunit.client.0.smithi083.stdout: 1 757310 15.92 MB/sec execute 138 sec latency 59.245 ms 2024-05-08T20:13:55.190 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:54 smithi083 ceph-mon[38695]: pgmap v435: 129 pgs: 129 active+clean; 179 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.7 MiB/s wr, 181 op/s 2024-05-08T20:13:55.232 INFO:tasks.workunit.client.1.smithi188.stdout: 1 441715 9.12 MB/sec execute 139 sec latency 9.607 ms 2024-05-08T20:13:55.340 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:54 smithi188 ceph-mon[48761]: pgmap v435: 129 pgs: 129 active+clean; 179 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.7 MiB/s wr, 181 op/s 2024-05-08T20:13:55.562 INFO:tasks.workunit.client.0.smithi083.stdout: 1 760679 15.94 MB/sec execute 139 sec latency 58.310 ms 2024-05-08T20:13:56.232 INFO:tasks.workunit.client.1.smithi188.stdout: 1 443622 9.15 MB/sec execute 140 sec latency 5.855 ms 2024-05-08T20:13:56.562 INFO:tasks.workunit.client.0.smithi083.stdout: 1 764210 15.96 MB/sec execute 140 sec latency 59.659 ms 2024-05-08T20:13:56.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:56 smithi188 ceph-mon[48761]: pgmap v436: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.6 MiB/s wr, 229 op/s 2024-05-08T20:13:56.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:56 smithi083 ceph-mon[38695]: pgmap v436: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.6 MiB/s wr, 229 op/s 2024-05-08T20:13:57.232 INFO:tasks.workunit.client.1.smithi188.stdout: 1 445174 9.12 MB/sec execute 141 sec latency 6.779 ms 2024-05-08T20:13:57.562 INFO:tasks.workunit.client.0.smithi083.stdout: 1 767689 15.98 MB/sec execute 141 sec latency 50.146 ms 2024-05-08T20:13:58.233 INFO:tasks.workunit.client.1.smithi188.stdout: 1 446969 9.15 MB/sec execute 142 sec latency 10.772 ms 2024-05-08T20:13:58.340 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:57 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:13:58.440 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:57 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:13:58.562 INFO:tasks.workunit.client.0.smithi083.stdout: 1 769955 15.96 MB/sec execute 142 sec latency 133.409 ms 2024-05-08T20:13:59.233 INFO:tasks.workunit.client.1.smithi188.stdout: 1 448483 9.13 MB/sec execute 143 sec latency 17.023 ms 2024-05-08T20:13:59.340 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:58 smithi188 ceph-mon[48761]: pgmap v437: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 199 op/s 2024-05-08T20:13:59.340 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:58 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:13:59.340 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:58 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:13:59.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:13:58 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:13:59.440 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:58 smithi083 ceph-mon[38695]: pgmap v437: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 199 op/s 2024-05-08T20:13:59.440 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:58 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:13:59.440 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:58 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:13:59.440 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:13:58 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:13:59.562 INFO:tasks.workunit.client.0.smithi083.stdout: 1 772917 15.94 MB/sec execute 143 sec latency 58.429 ms 2024-05-08T20:14:00.233 INFO:tasks.workunit.client.1.smithi188.stdout: 1 450305 9.13 MB/sec execute 144 sec latency 15.499 ms 2024-05-08T20:14:00.562 INFO:tasks.workunit.client.0.smithi083.stdout: 1 775536 15.95 MB/sec execute 144 sec latency 58.891 ms 2024-05-08T20:14:00.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:00 smithi188 ceph-mon[48761]: pgmap v438: 129 pgs: 129 active+clean; 185 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.6 MiB/s wr, 221 op/s 2024-05-08T20:14:00.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:00 smithi083 ceph-mon[38695]: pgmap v438: 129 pgs: 129 active+clean; 185 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.6 MiB/s wr, 221 op/s 2024-05-08T20:14:01.233 INFO:tasks.workunit.client.1.smithi188.stdout: 1 451903 9.13 MB/sec execute 145 sec latency 7.059 ms 2024-05-08T20:14:01.562 INFO:tasks.workunit.client.0.smithi083.stdout: 1 778517 15.94 MB/sec execute 145 sec latency 57.530 ms 2024-05-08T20:14:02.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 453789 9.14 MB/sec execute 146 sec latency 11.824 ms 2024-05-08T20:14:02.340 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:02 smithi188 ceph-mon[48761]: pgmap v439: 129 pgs: 129 active+clean; 186 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 183 op/s 2024-05-08T20:14:02.562 INFO:tasks.workunit.client.0.smithi083.stdout: 1 782022 15.96 MB/sec execute 146 sec latency 49.546 ms 2024-05-08T20:14:02.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:02 smithi083 ceph-mon[38695]: pgmap v439: 129 pgs: 129 active+clean; 186 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 183 op/s 2024-05-08T20:14:03.233 INFO:tasks.workunit.client.1.smithi188.stdout: 1 455158 9.12 MB/sec execute 147 sec latency 13.053 ms 2024-05-08T20:14:03.562 INFO:tasks.workunit.client.0.smithi083.stdout: 1 785491 15.98 MB/sec execute 147 sec latency 49.504 ms 2024-05-08T20:14:04.233 INFO:tasks.workunit.client.1.smithi188.stdout: 1 457010 9.13 MB/sec execute 148 sec latency 18.364 ms 2024-05-08T20:14:04.562 INFO:tasks.workunit.client.0.smithi083.stdout: 1 788971 16.00 MB/sec execute 148 sec latency 70.008 ms 2024-05-08T20:14:04.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:04 smithi188 ceph-mon[48761]: pgmap v440: 129 pgs: 129 active+clean; 186 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 3.8 MiB/s wr, 184 op/s 2024-05-08T20:14:04.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:04 smithi083 ceph-mon[38695]: pgmap v440: 129 pgs: 129 active+clean; 186 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 3.8 MiB/s wr, 184 op/s 2024-05-08T20:14:05.233 INFO:tasks.workunit.client.1.smithi188.stdout: 1 458344 9.11 MB/sec execute 149 sec latency 59.378 ms 2024-05-08T20:14:05.562 INFO:tasks.workunit.client.0.smithi083.stdout: 1 792461 16.02 MB/sec execute 149 sec latency 66.506 ms 2024-05-08T20:14:06.233 INFO:tasks.workunit.client.1.smithi188.stdout: 1 460206 9.23 MB/sec execute 150 sec latency 17.086 ms 2024-05-08T20:14:06.563 INFO:tasks.workunit.client.0.smithi083.stdout: 1 796004 16.04 MB/sec execute 150 sec latency 54.812 ms 2024-05-08T20:14:06.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:06 smithi188 ceph-mon[48761]: pgmap v441: 129 pgs: 129 active+clean; 188 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.4 MiB/s wr, 202 op/s 2024-05-08T20:14:06.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:06 smithi083 ceph-mon[38695]: pgmap v441: 129 pgs: 129 active+clean; 188 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.4 MiB/s wr, 202 op/s 2024-05-08T20:14:07.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 462040 9.25 MB/sec execute 151 sec latency 12.043 ms 2024-05-08T20:14:07.563 INFO:tasks.workunit.client.0.smithi083.stdout: 1 799548 16.06 MB/sec execute 151 sec latency 46.323 ms 2024-05-08T20:14:08.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 463437 9.23 MB/sec execute 152 sec latency 21.886 ms 2024-05-08T20:14:08.440 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:08 smithi083 ceph-mon[38695]: pgmap v442: 129 pgs: 129 active+clean; 190 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.1 MiB/s wr, 189 op/s 2024-05-08T20:14:08.563 INFO:tasks.workunit.client.0.smithi083.stdout: 1 803090 16.08 MB/sec execute 152 sec latency 58.312 ms 2024-05-08T20:14:08.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:08 smithi188 ceph-mon[48761]: pgmap v442: 129 pgs: 129 active+clean; 190 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.1 MiB/s wr, 189 op/s 2024-05-08T20:14:09.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 465231 9.23 MB/sec execute 153 sec latency 5.675 ms 2024-05-08T20:14:09.563 INFO:tasks.workunit.client.0.smithi083.stdout: 1 805651 16.06 MB/sec execute 153 sec latency 54.085 ms 2024-05-08T20:14:09.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:14:09.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:14:10.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 466657 9.22 MB/sec execute 154 sec latency 14.766 ms 2024-05-08T20:14:10.563 INFO:tasks.workunit.client.0.smithi083.stdout: 1 809217 16.09 MB/sec execute 154 sec latency 57.739 ms 2024-05-08T20:14:10.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:10 smithi188 ceph-mon[48761]: pgmap v443: 129 pgs: 129 active+clean; 194 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 9.5 KiB/s rd, 6.2 MiB/s wr, 222 op/s 2024-05-08T20:14:10.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:10 smithi083 ceph-mon[38695]: pgmap v443: 129 pgs: 129 active+clean; 194 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 9.5 KiB/s rd, 6.2 MiB/s wr, 222 op/s 2024-05-08T20:14:11.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 468655 9.23 MB/sec execute 155 sec latency 10.262 ms 2024-05-08T20:14:11.563 INFO:tasks.workunit.client.0.smithi083.stdout: 1 812828 16.11 MB/sec execute 155 sec latency 50.105 ms 2024-05-08T20:14:12.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 469979 9.21 MB/sec execute 156 sec latency 10.249 ms 2024-05-08T20:14:12.563 INFO:tasks.workunit.client.0.smithi083.stdout: 1 815964 16.12 MB/sec execute 156 sec latency 91.777 ms 2024-05-08T20:14:12.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:12 smithi188 ceph-mon[48761]: pgmap v444: 129 pgs: 129 active+clean; 191 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.0 MiB/s wr, 166 op/s 2024-05-08T20:14:12.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:12 smithi083 ceph-mon[38695]: pgmap v444: 129 pgs: 129 active+clean; 191 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.0 MiB/s wr, 166 op/s 2024-05-08T20:14:13.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 471756 9.22 MB/sec execute 157 sec latency 7.865 ms 2024-05-08T20:14:13.563 INFO:tasks.workunit.client.0.smithi083.stdout: 1 818971 16.10 MB/sec execute 157 sec latency 58.468 ms 2024-05-08T20:14:14.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 473274 9.21 MB/sec execute 158 sec latency 6.858 ms 2024-05-08T20:14:14.564 INFO:tasks.workunit.client.0.smithi083.stdout: 1 821775 16.11 MB/sec execute 158 sec latency 66.786 ms 2024-05-08T20:14:14.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:14 smithi188 ceph-mon[48761]: pgmap v445: 129 pgs: 129 active+clean; 194 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 9.5 KiB/s rd, 5.6 MiB/s wr, 220 op/s 2024-05-08T20:14:14.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:14 smithi083 ceph-mon[38695]: pgmap v445: 129 pgs: 129 active+clean; 194 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 9.5 KiB/s rd, 5.6 MiB/s wr, 220 op/s 2024-05-08T20:14:15.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 474972 9.20 MB/sec execute 159 sec latency 7.256 ms 2024-05-08T20:14:15.563 INFO:tasks.workunit.client.0.smithi083.stdout: 1 824610 16.10 MB/sec execute 159 sec latency 58.445 ms 2024-05-08T20:14:16.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 476635 9.22 MB/sec execute 160 sec latency 10.325 ms 2024-05-08T20:14:16.563 INFO:tasks.workunit.client.0.smithi083.stdout: 1 828037 16.12 MB/sec execute 160 sec latency 55.958 ms 2024-05-08T20:14:16.567 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:16 smithi083 ceph-mon[38695]: pgmap v446: 129 pgs: 129 active+clean; 189 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6.6 KiB/s rd, 6.0 MiB/s wr, 221 op/s 2024-05-08T20:14:16.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:16 smithi188 ceph-mon[48761]: pgmap v446: 129 pgs: 129 active+clean; 189 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6.6 KiB/s rd, 6.0 MiB/s wr, 221 op/s 2024-05-08T20:14:17.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 478435 9.20 MB/sec execute 161 sec latency 11.155 ms 2024-05-08T20:14:17.563 INFO:tasks.workunit.client.0.smithi083.stdout: 1 831520 16.13 MB/sec execute 161 sec latency 67.107 ms 2024-05-08T20:14:18.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 480088 9.22 MB/sec execute 162 sec latency 9.493 ms 2024-05-08T20:14:18.563 INFO:tasks.workunit.client.0.smithi083.stdout: 1 834997 16.15 MB/sec execute 162 sec latency 118.708 ms 2024-05-08T20:14:19.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 481729 9.20 MB/sec execute 163 sec latency 9.906 ms 2024-05-08T20:14:19.563 INFO:tasks.workunit.client.0.smithi083.stdout: 1 835004 16.05 MB/sec execute 163 sec latency 441.825 ms 2024-05-08T20:14:19.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:19 smithi188 ceph-mon[48761]: pgmap v447: 129 pgs: 129 active+clean; 190 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.2 MiB/s wr, 193 op/s 2024-05-08T20:14:19.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:19 smithi083 ceph-mon[38695]: pgmap v447: 129 pgs: 129 active+clean; 190 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.2 MiB/s wr, 193 op/s 2024-05-08T20:14:20.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 483521 9.22 MB/sec execute 164 sec latency 7.719 ms 2024-05-08T20:14:20.563 INFO:tasks.workunit.client.0.smithi083.stdout: 1 836775 16.00 MB/sec execute 164 sec latency 358.612 ms 2024-05-08T20:14:21.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 485092 9.20 MB/sec execute 165 sec latency 15.822 ms 2024-05-08T20:14:21.563 INFO:tasks.workunit.client.0.smithi083.stdout: 1 839798 16.01 MB/sec execute 165 sec latency 58.418 ms 2024-05-08T20:14:21.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:21 smithi188 ceph-mon[48761]: pgmap v448: 129 pgs: 129 active+clean; 194 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 9.4 KiB/s rd, 5.9 MiB/s wr, 218 op/s 2024-05-08T20:14:21.632 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:21 smithi083 ceph-mon[38695]: pgmap v448: 129 pgs: 129 active+clean; 194 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 9.4 KiB/s rd, 5.9 MiB/s wr, 218 op/s 2024-05-08T20:14:22.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 486974 9.23 MB/sec execute 166 sec latency 12.312 ms 2024-05-08T20:14:22.563 INFO:tasks.workunit.client.0.smithi083.stdout: 1 842438 16.00 MB/sec execute 166 sec latency 66.333 ms 2024-05-08T20:14:23.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 488454 9.20 MB/sec execute 167 sec latency 6.872 ms 2024-05-08T20:14:23.563 INFO:tasks.workunit.client.0.smithi083.stdout: 1 845830 16.01 MB/sec execute 167 sec latency 58.740 ms 2024-05-08T20:14:23.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:23 smithi188 ceph-mon[48761]: pgmap v449: 129 pgs: 129 active+clean; 194 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.5 MiB/s wr, 179 op/s 2024-05-08T20:14:23.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:23 smithi083 ceph-mon[38695]: pgmap v449: 129 pgs: 129 active+clean; 194 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.5 MiB/s wr, 179 op/s 2024-05-08T20:14:24.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 490317 9.22 MB/sec execute 168 sec latency 13.032 ms 2024-05-08T20:14:24.563 INFO:tasks.workunit.client.0.smithi083.stdout: 1 849364 16.03 MB/sec execute 168 sec latency 57.150 ms 2024-05-08T20:14:24.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:24 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:14:24.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:14:25.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 491802 9.21 MB/sec execute 169 sec latency 15.426 ms 2024-05-08T20:14:25.563 INFO:tasks.workunit.client.0.smithi083.stdout: 1 852799 16.05 MB/sec execute 169 sec latency 75.117 ms 2024-05-08T20:14:25.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:25 smithi188 ceph-mon[48761]: pgmap v450: 129 pgs: 129 active+clean; 195 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.8 KiB/s rd, 4.2 MiB/s wr, 214 op/s 2024-05-08T20:14:25.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:25 smithi083 ceph-mon[38695]: pgmap v450: 129 pgs: 129 active+clean; 195 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.8 KiB/s rd, 4.2 MiB/s wr, 214 op/s 2024-05-08T20:14:26.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 493635 9.21 MB/sec execute 170 sec latency 10.605 ms 2024-05-08T20:14:26.563 INFO:tasks.workunit.client.0.smithi083.stdout: 1 855447 16.04 MB/sec execute 170 sec latency 65.280 ms 2024-05-08T20:14:27.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 495074 9.19 MB/sec execute 171 sec latency 15.212 ms 2024-05-08T20:14:27.564 INFO:tasks.workunit.client.0.smithi083.stdout: 1 858573 16.05 MB/sec execute 171 sec latency 59.664 ms 2024-05-08T20:14:27.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:27 smithi188 ceph-mon[48761]: pgmap v451: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.4 MiB/s wr, 203 op/s 2024-05-08T20:14:27.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:27 smithi083 ceph-mon[38695]: pgmap v451: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.4 MiB/s wr, 203 op/s 2024-05-08T20:14:28.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 497052 9.21 MB/sec execute 172 sec latency 7.932 ms 2024-05-08T20:14:28.564 INFO:tasks.workunit.client.0.smithi083.stdout: 1 861549 16.03 MB/sec execute 172 sec latency 51.579 ms 2024-05-08T20:14:29.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 498418 9.19 MB/sec execute 173 sec latency 9.322 ms 2024-05-08T20:14:29.564 INFO:tasks.workunit.client.0.smithi083.stdout: 1 864232 16.04 MB/sec execute 173 sec latency 51.318 ms 2024-05-08T20:14:29.590 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:29 smithi188 ceph-mon[48761]: pgmap v452: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.9 MiB/s wr, 191 op/s 2024-05-08T20:14:29.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:29 smithi083 ceph-mon[38695]: pgmap v452: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.9 MiB/s wr, 191 op/s 2024-05-08T20:14:30.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 500295 9.20 MB/sec execute 174 sec latency 24.112 ms 2024-05-08T20:14:30.564 INFO:tasks.workunit.client.0.smithi083.stdout: 1 867095 16.03 MB/sec execute 174 sec latency 105.081 ms 2024-05-08T20:14:31.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 501805 9.19 MB/sec execute 175 sec latency 10.531 ms 2024-05-08T20:14:31.564 INFO:tasks.workunit.client.0.smithi083.stdout: 1 870570 16.05 MB/sec execute 175 sec latency 62.154 ms 2024-05-08T20:14:31.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:31 smithi188 ceph-mon[48761]: pgmap v453: 129 pgs: 129 active+clean; 191 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.8 MiB/s wr, 244 op/s 2024-05-08T20:14:31.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:31 smithi083 ceph-mon[38695]: pgmap v453: 129 pgs: 129 active+clean; 191 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.8 MiB/s wr, 244 op/s 2024-05-08T20:14:32.235 INFO:tasks.workunit.client.1.smithi188.stdout: 1 503501 9.18 MB/sec execute 176 sec latency 15.531 ms 2024-05-08T20:14:32.564 INFO:tasks.workunit.client.0.smithi083.stdout: 1 874066 16.06 MB/sec execute 176 sec latency 46.599 ms 2024-05-08T20:14:33.234 INFO:tasks.workunit.client.1.smithi188.stdout: 1 505113 9.19 MB/sec execute 177 sec latency 6.594 ms 2024-05-08T20:14:33.564 INFO:tasks.workunit.client.0.smithi083.stdout: 1 876579 16.05 MB/sec execute 177 sec latency 55.774 ms 2024-05-08T20:14:33.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:33 smithi188 ceph-mon[48761]: pgmap v454: 129 pgs: 129 active+clean; 194 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.6 MiB/s wr, 197 op/s 2024-05-08T20:14:33.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:33 smithi083 ceph-mon[38695]: pgmap v454: 129 pgs: 129 active+clean; 194 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.6 MiB/s wr, 197 op/s 2024-05-08T20:14:34.235 INFO:tasks.workunit.client.1.smithi188.stdout: 1 506856 9.18 MB/sec execute 178 sec latency 19.183 ms 2024-05-08T20:14:34.564 INFO:tasks.workunit.client.0.smithi083.stdout: 1 879304 16.03 MB/sec execute 178 sec latency 49.372 ms 2024-05-08T20:14:35.235 INFO:tasks.workunit.client.1.smithi188.stdout: 1 508502 9.20 MB/sec execute 179 sec latency 11.583 ms 2024-05-08T20:14:35.564 INFO:tasks.workunit.client.0.smithi083.stdout: 1 881964 16.03 MB/sec execute 179 sec latency 49.361 ms 2024-05-08T20:14:35.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:35 smithi188 ceph-mon[48761]: pgmap v455: 129 pgs: 129 active+clean; 193 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.2 MiB/s wr, 214 op/s 2024-05-08T20:14:35.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:35 smithi083 ceph-mon[38695]: pgmap v455: 129 pgs: 129 active+clean; 193 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.2 MiB/s wr, 214 op/s 2024-05-08T20:14:36.235 INFO:tasks.workunit.client.1.smithi188.stdout: 1 510213 9.18 MB/sec execute 180 sec latency 27.436 ms 2024-05-08T20:14:36.564 INFO:tasks.workunit.client.0.smithi083.stdout: 1 884901 16.03 MB/sec execute 180 sec latency 56.073 ms 2024-05-08T20:14:37.235 INFO:tasks.workunit.client.1.smithi188.stdout: 1 511958 9.20 MB/sec execute 181 sec latency 7.285 ms 2024-05-08T20:14:37.542 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:37 smithi188 ceph-mon[48761]: pgmap v456: 129 pgs: 129 active+clean; 195 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.2 MiB/s wr, 185 op/s 2024-05-08T20:14:37.564 INFO:tasks.workunit.client.0.smithi083.stdout: 1 888311 16.04 MB/sec execute 181 sec latency 70.552 ms 2024-05-08T20:14:37.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:37 smithi083 ceph-mon[38695]: pgmap v456: 129 pgs: 129 active+clean; 195 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.2 MiB/s wr, 185 op/s 2024-05-08T20:14:38.235 INFO:tasks.workunit.client.1.smithi188.stdout: 1 513598 9.18 MB/sec execute 182 sec latency 10.565 ms 2024-05-08T20:14:38.564 INFO:tasks.workunit.client.0.smithi083.stdout: 1 891795 16.06 MB/sec execute 182 sec latency 61.209 ms 2024-05-08T20:14:39.235 INFO:tasks.workunit.client.1.smithi188.stdout: 1 515412 9.20 MB/sec execute 183 sec latency 9.650 ms 2024-05-08T20:14:39.564 INFO:tasks.workunit.client.0.smithi083.stdout: 1 894445 16.05 MB/sec execute 183 sec latency 61.378 ms 2024-05-08T20:14:39.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:39 smithi188 ceph-mon[48761]: pgmap v457: 129 pgs: 129 active+clean; 192 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.9 MiB/s wr, 180 op/s 2024-05-08T20:14:39.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:14:39.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:39 smithi083 ceph-mon[38695]: pgmap v457: 129 pgs: 129 active+clean; 192 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.9 MiB/s wr, 180 op/s 2024-05-08T20:14:39.691 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:14:40.235 INFO:tasks.workunit.client.1.smithi188.stdout: 1 516952 9.19 MB/sec execute 184 sec latency 12.190 ms 2024-05-08T20:14:40.564 INFO:tasks.workunit.client.0.smithi083.stdout: 1 897637 16.05 MB/sec execute 184 sec latency 58.393 ms 2024-05-08T20:14:41.235 INFO:tasks.workunit.client.1.smithi188.stdout: 1 518796 9.20 MB/sec execute 185 sec latency 18.794 ms 2024-05-08T20:14:41.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 900093 16.03 MB/sec execute 185 sec latency 50.166 ms 2024-05-08T20:14:41.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:41 smithi188 ceph-mon[48761]: pgmap v458: 129 pgs: 129 active+clean; 194 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.7 MiB/s wr, 239 op/s 2024-05-08T20:14:41.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:41 smithi083 ceph-mon[38695]: pgmap v458: 129 pgs: 129 active+clean; 194 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.7 MiB/s wr, 239 op/s 2024-05-08T20:14:42.235 INFO:tasks.workunit.client.1.smithi188.stdout: 1 520268 9.19 MB/sec execute 186 sec latency 10.758 ms 2024-05-08T20:14:42.564 INFO:tasks.workunit.client.0.smithi083.stdout: 1 902738 16.03 MB/sec execute 186 sec latency 66.583 ms 2024-05-08T20:14:43.235 INFO:tasks.workunit.client.1.smithi188.stdout: 1 522028 9.19 MB/sec execute 187 sec latency 19.198 ms 2024-05-08T20:14:43.564 INFO:tasks.workunit.client.0.smithi083.stdout: 1 906103 16.04 MB/sec execute 187 sec latency 46.421 ms 2024-05-08T20:14:43.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:43 smithi188 ceph-mon[48761]: pgmap v459: 129 pgs: 129 active+clean; 192 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.6 MiB/s wr, 180 op/s 2024-05-08T20:14:43.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:43 smithi083 ceph-mon[38695]: pgmap v459: 129 pgs: 129 active+clean; 192 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.6 MiB/s wr, 180 op/s 2024-05-08T20:14:44.235 INFO:tasks.workunit.client.1.smithi188.stdout: 1 523498 9.18 MB/sec execute 188 sec latency 8.770 ms 2024-05-08T20:14:44.564 INFO:tasks.workunit.client.0.smithi083.stdout: 1 909583 16.06 MB/sec execute 188 sec latency 50.038 ms 2024-05-08T20:14:45.235 INFO:tasks.workunit.client.1.smithi188.stdout: 1 525470 9.19 MB/sec execute 189 sec latency 16.529 ms 2024-05-08T20:14:45.564 INFO:tasks.workunit.client.0.smithi083.stdout: 1 913067 16.07 MB/sec execute 189 sec latency 50.001 ms 2024-05-08T20:14:45.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:45 smithi188 ceph-mon[48761]: pgmap v460: 129 pgs: 129 active+clean; 192 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.0 MiB/s wr, 178 op/s 2024-05-08T20:14:45.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:45 smithi083 ceph-mon[38695]: pgmap v460: 129 pgs: 129 active+clean; 192 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.0 MiB/s wr, 178 op/s 2024-05-08T20:14:46.235 INFO:tasks.workunit.client.1.smithi188.stdout: 1 526749 9.17 MB/sec execute 190 sec latency 11.139 ms 2024-05-08T20:14:46.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 915708 16.06 MB/sec execute 190 sec latency 49.755 ms 2024-05-08T20:14:47.235 INFO:tasks.workunit.client.1.smithi188.stdout: 1 528521 9.18 MB/sec execute 191 sec latency 10.785 ms 2024-05-08T20:14:47.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 918576 16.14 MB/sec execute 191 sec latency 53.328 ms 2024-05-08T20:14:47.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:47 smithi188 ceph-mon[48761]: pgmap v461: 129 pgs: 129 active+clean; 188 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.2 MiB/s wr, 194 op/s 2024-05-08T20:14:47.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:47 smithi083 ceph-mon[38695]: pgmap v461: 129 pgs: 129 active+clean; 188 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.2 MiB/s wr, 194 op/s 2024-05-08T20:14:48.235 INFO:tasks.workunit.client.1.smithi188.stdout: 1 530024 9.17 MB/sec execute 192 sec latency 10.181 ms 2024-05-08T20:14:48.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 921240 16.13 MB/sec execute 192 sec latency 58.056 ms 2024-05-08T20:14:49.235 INFO:tasks.workunit.client.1.smithi188.stdout: 1 531728 9.16 MB/sec execute 193 sec latency 8.720 ms 2024-05-08T20:14:49.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 924599 16.15 MB/sec execute 193 sec latency 56.721 ms 2024-05-08T20:14:49.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:49 smithi188 ceph-mon[48761]: pgmap v462: 129 pgs: 129 active+clean; 190 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.8 KiB/s rd, 4.2 MiB/s wr, 228 op/s 2024-05-08T20:14:49.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:49 smithi083 ceph-mon[38695]: pgmap v462: 129 pgs: 129 active+clean; 190 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.8 KiB/s rd, 4.2 MiB/s wr, 228 op/s 2024-05-08T20:14:50.235 INFO:tasks.workunit.client.1.smithi188.stdout: 1 533386 9.18 MB/sec execute 194 sec latency 8.108 ms 2024-05-08T20:14:50.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 928081 16.16 MB/sec execute 194 sec latency 49.779 ms 2024-05-08T20:14:51.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 535110 9.16 MB/sec execute 195 sec latency 20.644 ms 2024-05-08T20:14:51.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 928220 16.08 MB/sec execute 195 sec latency 409.933 ms 2024-05-08T20:14:51.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:51 smithi188 ceph-mon[48761]: pgmap v463: 129 pgs: 129 active+clean; 192 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.4 MiB/s wr, 218 op/s 2024-05-08T20:14:51.690 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:51 smithi083 ceph-mon[38695]: pgmap v463: 129 pgs: 129 active+clean; 192 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.4 MiB/s wr, 218 op/s 2024-05-08T20:14:52.235 INFO:tasks.workunit.client.1.smithi188.stdout: 1 536810 9.18 MB/sec execute 196 sec latency 25.177 ms 2024-05-08T20:14:52.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 930320 16.06 MB/sec execute 196 sec latency 364.892 ms 2024-05-08T20:14:53.235 INFO:tasks.workunit.client.1.smithi188.stdout: 1 538417 9.16 MB/sec execute 197 sec latency 10.711 ms 2024-05-08T20:14:53.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 932773 16.04 MB/sec execute 197 sec latency 74.795 ms 2024-05-08T20:14:53.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:53 smithi188 ceph-mon[48761]: pgmap v464: 129 pgs: 129 active+clean; 192 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.5 MiB/s wr, 157 op/s 2024-05-08T20:14:53.691 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:53 smithi083 ceph-mon[38695]: pgmap v464: 129 pgs: 129 active+clean; 192 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.5 MiB/s wr, 157 op/s 2024-05-08T20:14:54.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 540246 9.18 MB/sec execute 198 sec latency 13.165 ms 2024-05-08T20:14:54.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 935424 16.03 MB/sec execute 198 sec latency 50.368 ms 2024-05-08T20:14:54.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:14:54.691 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:14:55.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 541789 9.17 MB/sec execute 199 sec latency 11.343 ms 2024-05-08T20:14:55.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 938822 16.04 MB/sec execute 199 sec latency 50.423 ms 2024-05-08T20:14:55.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:55 smithi188 ceph-mon[48761]: pgmap v465: 129 pgs: 129 active+clean; 191 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.8 KiB/s rd, 4.0 MiB/s wr, 178 op/s 2024-05-08T20:14:55.691 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:55 smithi083 ceph-mon[38695]: pgmap v465: 129 pgs: 129 active+clean; 191 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.8 KiB/s rd, 4.0 MiB/s wr, 178 op/s 2024-05-08T20:14:56.235 INFO:tasks.workunit.client.1.smithi188.stdout: 1 543639 9.18 MB/sec execute 200 sec latency 6.921 ms 2024-05-08T20:14:56.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 942304 16.06 MB/sec execute 200 sec latency 59.013 ms 2024-05-08T20:14:57.235 INFO:tasks.workunit.client.1.smithi188.stdout: 1 545080 9.17 MB/sec execute 201 sec latency 12.075 ms 2024-05-08T20:14:57.441 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:57 smithi083 ceph-mon[38695]: pgmap v466: 129 pgs: 129 active+clean; 191 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.3 MiB/s wr, 212 op/s 2024-05-08T20:14:57.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 945786 16.07 MB/sec execute 201 sec latency 57.730 ms 2024-05-08T20:14:57.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:57 smithi188 ceph-mon[48761]: pgmap v466: 129 pgs: 129 active+clean; 191 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.3 MiB/s wr, 212 op/s 2024-05-08T20:14:58.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 546853 9.17 MB/sec execute 202 sec latency 12.170 ms 2024-05-08T20:14:58.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 948432 16.06 MB/sec execute 202 sec latency 58.429 ms 2024-05-08T20:14:59.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 548225 9.16 MB/sec execute 203 sec latency 7.531 ms 2024-05-08T20:14:59.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 951572 16.07 MB/sec execute 203 sec latency 58.386 ms 2024-05-08T20:14:59.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:59 smithi188 ceph-mon[48761]: pgmap v467: 129 pgs: 129 active+clean; 193 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.8 KiB/s rd, 4.1 MiB/s wr, 193 op/s 2024-05-08T20:14:59.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:14:59.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:14:59.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:14:59.592 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:14:59.592 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:14:59.592 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:59 smithi188 ceph-mon[48761]: Adjusting osd_memory_target on smithi083 to 920.6M 2024-05-08T20:14:59.592 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:14:59.592 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:14:59.592 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:14:59.592 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:14:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:14:59.691 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:59 smithi083 ceph-mon[38695]: pgmap v467: 129 pgs: 129 active+clean; 193 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.8 KiB/s rd, 4.1 MiB/s wr, 193 op/s 2024-05-08T20:14:59.691 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:14:59.691 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:14:59.691 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:14:59.691 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:14:59.691 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:14:59.691 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:59 smithi083 ceph-mon[38695]: Adjusting osd_memory_target on smithi083 to 920.6M 2024-05-08T20:14:59.691 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:14:59.692 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:14:59.692 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:14:59.692 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:14:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:15:00.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 550060 9.16 MB/sec execute 204 sec latency 12.590 ms 2024-05-08T20:15:00.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 954088 16.05 MB/sec execute 204 sec latency 57.250 ms 2024-05-08T20:15:01.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 551504 9.16 MB/sec execute 205 sec latency 8.498 ms 2024-05-08T20:15:01.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 956733 16.04 MB/sec execute 205 sec latency 56.579 ms 2024-05-08T20:15:01.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:01 smithi188 ceph-mon[48761]: pgmap v468: 129 pgs: 129 active+clean; 191 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.3 MiB/s wr, 200 op/s 2024-05-08T20:15:01.691 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:01 smithi083 ceph-mon[38695]: pgmap v468: 129 pgs: 129 active+clean; 191 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.3 MiB/s wr, 200 op/s 2024-05-08T20:15:02.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 553173 9.15 MB/sec execute 206 sec latency 14.004 ms 2024-05-08T20:15:02.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 960091 16.06 MB/sec execute 206 sec latency 82.976 ms 2024-05-08T20:15:03.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 554769 9.16 MB/sec execute 207 sec latency 5.518 ms 2024-05-08T20:15:03.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 963524 16.07 MB/sec execute 207 sec latency 141.661 ms 2024-05-08T20:15:03.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:03 smithi188 ceph-mon[48761]: pgmap v469: 129 pgs: 129 active+clean; 191 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.5 MiB/s wr, 172 op/s 2024-05-08T20:15:03.691 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:03 smithi083 ceph-mon[38695]: pgmap v469: 129 pgs: 129 active+clean; 191 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.5 MiB/s wr, 172 op/s 2024-05-08T20:15:04.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 556489 9.14 MB/sec execute 208 sec latency 21.055 ms 2024-05-08T20:15:04.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 966169 16.06 MB/sec execute 208 sec latency 58.503 ms 2024-05-08T20:15:05.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 558128 9.16 MB/sec execute 209 sec latency 12.705 ms 2024-05-08T20:15:05.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 969357 16.07 MB/sec execute 209 sec latency 58.418 ms 2024-05-08T20:15:05.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:05 smithi188 ceph-mon[48761]: pgmap v470: 129 pgs: 129 active+clean; 190 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.8 KiB/s rd, 3.9 MiB/s wr, 183 op/s 2024-05-08T20:15:05.691 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:05 smithi083 ceph-mon[38695]: pgmap v470: 129 pgs: 129 active+clean; 190 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.8 KiB/s rd, 3.9 MiB/s wr, 183 op/s 2024-05-08T20:15:06.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 559769 9.14 MB/sec execute 210 sec latency 9.194 ms 2024-05-08T20:15:06.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 972271 16.05 MB/sec execute 210 sec latency 58.430 ms 2024-05-08T20:15:07.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 561552 9.16 MB/sec execute 211 sec latency 4.285 ms 2024-05-08T20:15:07.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 974956 16.06 MB/sec execute 211 sec latency 58.443 ms 2024-05-08T20:15:07.591 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:07 smithi188 ceph-mon[48761]: pgmap v471: 129 pgs: 129 active+clean; 190 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.5 MiB/s wr, 223 op/s 2024-05-08T20:15:07.691 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:07 smithi083 ceph-mon[38695]: pgmap v471: 129 pgs: 129 active+clean; 190 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.5 MiB/s wr, 223 op/s 2024-05-08T20:15:08.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 563066 9.15 MB/sec execute 212 sec latency 14.215 ms 2024-05-08T20:15:08.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 977900 16.05 MB/sec execute 212 sec latency 58.396 ms 2024-05-08T20:15:09.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:15:09.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 564880 9.16 MB/sec execute 213 sec latency 7.162 ms 2024-05-08T20:15:09.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:15:09.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 981358 16.07 MB/sec execute 213 sec latency 68.890 ms 2024-05-08T20:15:10.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:09 smithi083 ceph-mon[38695]: pgmap v472: 129 pgs: 129 active+clean; 189 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 8.8 KiB/s rd, 3.8 MiB/s wr, 175 op/s 2024-05-08T20:15:10.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 566353 9.15 MB/sec execute 214 sec latency 17.812 ms 2024-05-08T20:15:10.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:09 smithi188 ceph-mon[48761]: pgmap v472: 129 pgs: 129 active+clean; 189 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 8.8 KiB/s rd, 3.8 MiB/s wr, 175 op/s 2024-05-08T20:15:10.565 INFO:tasks.workunit.client.0.smithi083.stdout: 1 984797 16.08 MB/sec execute 214 sec latency 61.823 ms 2024-05-08T20:15:11.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 568137 9.15 MB/sec execute 215 sec latency 15.490 ms 2024-05-08T20:15:11.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 987443 16.07 MB/sec execute 215 sec latency 61.962 ms 2024-05-08T20:15:12.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:11 smithi083 ceph-mon[38695]: pgmap v473: 129 pgs: 129 active+clean; 192 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.4 MiB/s wr, 224 op/s 2024-05-08T20:15:12.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 569636 9.14 MB/sec execute 216 sec latency 15.877 ms 2024-05-08T20:15:12.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:11 smithi188 ceph-mon[48761]: pgmap v473: 129 pgs: 129 active+clean; 192 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.4 MiB/s wr, 224 op/s 2024-05-08T20:15:12.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 990603 16.08 MB/sec execute 216 sec latency 58.066 ms 2024-05-08T20:15:13.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 571629 9.15 MB/sec execute 217 sec latency 18.496 ms 2024-05-08T20:15:13.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 993609 16.06 MB/sec execute 217 sec latency 57.939 ms 2024-05-08T20:15:14.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:13 smithi083 ceph-mon[38695]: pgmap v474: 129 pgs: 129 active+clean; 192 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.4 MiB/s wr, 166 op/s 2024-05-08T20:15:14.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 572984 9.14 MB/sec execute 218 sec latency 13.436 ms 2024-05-08T20:15:14.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:13 smithi188 ceph-mon[48761]: pgmap v474: 129 pgs: 129 active+clean; 192 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.4 MiB/s wr, 166 op/s 2024-05-08T20:15:14.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 996412 16.07 MB/sec execute 218 sec latency 61.710 ms 2024-05-08T20:15:15.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 574858 9.15 MB/sec execute 219 sec latency 12.087 ms 2024-05-08T20:15:15.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 999247 16.06 MB/sec execute 219 sec latency 58.432 ms 2024-05-08T20:15:16.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:15 smithi083 ceph-mon[38695]: pgmap v475: 129 pgs: 129 active+clean; 191 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.9 MiB/s wr, 177 op/s 2024-05-08T20:15:16.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 576341 9.14 MB/sec execute 220 sec latency 6.634 ms 2024-05-08T20:15:16.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:15 smithi188 ceph-mon[48761]: pgmap v475: 129 pgs: 129 active+clean; 191 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.9 MiB/s wr, 177 op/s 2024-05-08T20:15:16.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1002677 16.08 MB/sec execute 220 sec latency 58.430 ms 2024-05-08T20:15:17.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 578031 9.13 MB/sec execute 221 sec latency 13.244 ms 2024-05-08T20:15:17.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1006221 16.09 MB/sec execute 221 sec latency 53.691 ms 2024-05-08T20:15:18.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:17 smithi083 ceph-mon[38695]: pgmap v476: 129 pgs: 129 active+clean; 194 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.4 MiB/s wr, 225 op/s 2024-05-08T20:15:18.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 579659 9.14 MB/sec execute 222 sec latency 13.396 ms 2024-05-08T20:15:18.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:17 smithi188 ceph-mon[48761]: pgmap v476: 129 pgs: 129 active+clean; 194 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.4 MiB/s wr, 225 op/s 2024-05-08T20:15:18.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1009636 16.10 MB/sec execute 222 sec latency 249.515 ms 2024-05-08T20:15:19.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 581372 9.13 MB/sec execute 223 sec latency 8.798 ms 2024-05-08T20:15:19.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1012252 16.09 MB/sec execute 223 sec latency 56.804 ms 2024-05-08T20:15:20.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:19 smithi083 ceph-mon[38695]: pgmap v477: 129 pgs: 129 active+clean; 191 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.9 MiB/s wr, 175 op/s 2024-05-08T20:15:20.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 582997 9.14 MB/sec execute 224 sec latency 9.837 ms 2024-05-08T20:15:20.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:19 smithi188 ceph-mon[48761]: pgmap v477: 129 pgs: 129 active+clean; 191 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.9 MiB/s wr, 175 op/s 2024-05-08T20:15:20.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1015700 16.10 MB/sec execute 224 sec latency 57.481 ms 2024-05-08T20:15:21.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 584672 9.13 MB/sec execute 225 sec latency 21.386 ms 2024-05-08T20:15:21.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1019027 16.11 MB/sec execute 225 sec latency 58.360 ms 2024-05-08T20:15:22.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:21 smithi083 ceph-mon[38695]: pgmap v478: 129 pgs: 129 active+clean; 190 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.5 MiB/s wr, 228 op/s 2024-05-08T20:15:22.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 586393 9.15 MB/sec execute 226 sec latency 10.483 ms 2024-05-08T20:15:22.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:21 smithi188 ceph-mon[48761]: pgmap v478: 129 pgs: 129 active+clean; 190 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.5 MiB/s wr, 228 op/s 2024-05-08T20:15:22.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1022077 16.11 MB/sec execute 226 sec latency 57.749 ms 2024-05-08T20:15:23.236 INFO:tasks.workunit.client.1.smithi188.stdout: 1 587899 9.13 MB/sec execute 227 sec latency 12.155 ms 2024-05-08T20:15:23.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1023873 16.09 MB/sec execute 227 sec latency 276.955 ms 2024-05-08T20:15:24.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:23 smithi083 ceph-mon[38695]: pgmap v479: 129 pgs: 129 active+clean; 190 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.5 MiB/s wr, 166 op/s 2024-05-08T20:15:24.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:15:24.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 589734 9.15 MB/sec execute 228 sec latency 13.126 ms 2024-05-08T20:15:24.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:23 smithi188 ceph-mon[48761]: pgmap v479: 129 pgs: 129 active+clean; 190 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.5 MiB/s wr, 166 op/s 2024-05-08T20:15:24.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:15:24.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1023880 16.02 MB/sec execute 228 sec latency 346.660 ms 2024-05-08T20:15:25.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 591223 9.13 MB/sec execute 229 sec latency 17.915 ms 2024-05-08T20:15:25.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1027374 16.03 MB/sec execute 229 sec latency 336.977 ms 2024-05-08T20:15:26.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:25 smithi083 ceph-mon[38695]: pgmap v480: 129 pgs: 129 active+clean; 189 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.1 MiB/s wr, 181 op/s 2024-05-08T20:15:26.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 593007 9.13 MB/sec execute 230 sec latency 13.816 ms 2024-05-08T20:15:26.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:25 smithi188 ceph-mon[48761]: pgmap v480: 129 pgs: 129 active+clean; 189 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.1 MiB/s wr, 181 op/s 2024-05-08T20:15:26.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1030910 16.04 MB/sec execute 230 sec latency 55.843 ms 2024-05-08T20:15:27.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 594588 9.14 MB/sec execute 231 sec latency 8.373 ms 2024-05-08T20:15:27.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1033567 16.03 MB/sec execute 231 sec latency 55.948 ms 2024-05-08T20:15:28.131 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:27 smithi188 ceph-mon[48761]: pgmap v481: 129 pgs: 129 active+clean; 189 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 226 op/s 2024-05-08T20:15:28.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:27 smithi083 ceph-mon[38695]: pgmap v481: 129 pgs: 129 active+clean; 189 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 226 op/s 2024-05-08T20:15:28.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 596467 9.14 MB/sec execute 232 sec latency 22.819 ms 2024-05-08T20:15:28.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1037105 16.05 MB/sec execute 232 sec latency 58.500 ms 2024-05-08T20:15:29.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 597853 9.13 MB/sec execute 233 sec latency 7.903 ms 2024-05-08T20:15:29.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1040657 16.06 MB/sec execute 233 sec latency 58.453 ms 2024-05-08T20:15:30.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:29 smithi083 ceph-mon[38695]: pgmap v482: 129 pgs: 129 active+clean; 190 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.0 MiB/s wr, 178 op/s 2024-05-08T20:15:30.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 599695 9.13 MB/sec execute 234 sec latency 7.162 ms 2024-05-08T20:15:30.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:29 smithi188 ceph-mon[48761]: pgmap v482: 129 pgs: 129 active+clean; 190 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.0 MiB/s wr, 178 op/s 2024-05-08T20:15:30.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1044175 16.07 MB/sec execute 234 sec latency 49.363 ms 2024-05-08T20:15:31.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 601139 9.13 MB/sec execute 235 sec latency 6.136 ms 2024-05-08T20:15:31.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1047336 16.08 MB/sec execute 235 sec latency 51.515 ms 2024-05-08T20:15:32.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:31 smithi083 ceph-mon[38695]: pgmap v483: 129 pgs: 129 active+clean; 192 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 228 op/s 2024-05-08T20:15:32.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 602825 9.12 MB/sec execute 236 sec latency 11.638 ms 2024-05-08T20:15:32.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:31 smithi188 ceph-mon[48761]: pgmap v483: 129 pgs: 129 active+clean; 192 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 228 op/s 2024-05-08T20:15:32.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1050312 16.07 MB/sec execute 236 sec latency 52.521 ms 2024-05-08T20:15:33.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 604414 9.13 MB/sec execute 237 sec latency 12.652 ms 2024-05-08T20:15:33.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1052996 16.07 MB/sec execute 237 sec latency 53.450 ms 2024-05-08T20:15:34.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:33 smithi083 ceph-mon[38695]: pgmap v484: 129 pgs: 129 active+clean; 192 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.4 MiB/s wr, 166 op/s 2024-05-08T20:15:34.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 606130 9.12 MB/sec execute 238 sec latency 9.610 ms 2024-05-08T20:15:34.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:33 smithi188 ceph-mon[48761]: pgmap v484: 129 pgs: 129 active+clean; 192 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.4 MiB/s wr, 166 op/s 2024-05-08T20:15:34.566 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1055942 16.07 MB/sec execute 238 sec latency 49.733 ms 2024-05-08T20:15:35.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 607749 9.13 MB/sec execute 239 sec latency 9.388 ms 2024-05-08T20:15:35.567 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1059340 16.08 MB/sec execute 239 sec latency 150.119 ms 2024-05-08T20:15:36.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:35 smithi083 ceph-mon[38695]: pgmap v485: 129 pgs: 129 active+clean; 194 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.2 MiB/s wr, 204 op/s 2024-05-08T20:15:36.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 609304 9.11 MB/sec execute 240 sec latency 10.515 ms 2024-05-08T20:15:36.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:35 smithi188 ceph-mon[48761]: pgmap v485: 129 pgs: 129 active+clean; 194 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.2 MiB/s wr, 204 op/s 2024-05-08T20:15:36.567 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1062833 16.09 MB/sec execute 240 sec latency 66.727 ms 2024-05-08T20:15:37.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 611102 9.13 MB/sec execute 241 sec latency 12.139 ms 2024-05-08T20:15:37.567 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1065473 16.08 MB/sec execute 241 sec latency 60.460 ms 2024-05-08T20:15:38.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:37 smithi083 ceph-mon[38695]: pgmap v486: 129 pgs: 129 active+clean; 195 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.2 MiB/s wr, 207 op/s 2024-05-08T20:15:38.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 612543 9.12 MB/sec execute 242 sec latency 13.834 ms 2024-05-08T20:15:38.248 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:37 smithi188 ceph-mon[48761]: pgmap v486: 129 pgs: 129 active+clean; 195 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.2 MiB/s wr, 207 op/s 2024-05-08T20:15:38.567 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1068669 16.09 MB/sec execute 242 sec latency 53.560 ms 2024-05-08T20:15:39.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:15:39.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 614286 9.12 MB/sec execute 243 sec latency 4.581 ms 2024-05-08T20:15:39.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:15:39.567 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1071584 16.08 MB/sec execute 243 sec latency 53.313 ms 2024-05-08T20:15:40.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:39 smithi083 ceph-mon[38695]: pgmap v487: 129 pgs: 129 active+clean; 191 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.0 MiB/s wr, 192 op/s 2024-05-08T20:15:40.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 615847 9.12 MB/sec execute 244 sec latency 17.827 ms 2024-05-08T20:15:40.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:39 smithi188 ceph-mon[48761]: pgmap v487: 129 pgs: 129 active+clean; 191 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.0 MiB/s wr, 192 op/s 2024-05-08T20:15:40.567 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1073773 16.06 MB/sec execute 244 sec latency 54.629 ms 2024-05-08T20:15:41.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 617739 9.12 MB/sec execute 245 sec latency 8.095 ms 2024-05-08T20:15:41.567 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1077140 16.08 MB/sec execute 245 sec latency 58.435 ms 2024-05-08T20:15:42.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:41 smithi083 ceph-mon[38695]: pgmap v488: 129 pgs: 129 active+clean; 191 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.2 MiB/s wr, 201 op/s 2024-05-08T20:15:42.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 619107 9.11 MB/sec execute 246 sec latency 15.116 ms 2024-05-08T20:15:42.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:41 smithi188 ceph-mon[48761]: pgmap v488: 129 pgs: 129 active+clean; 191 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.2 MiB/s wr, 201 op/s 2024-05-08T20:15:42.567 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1080672 16.09 MB/sec execute 246 sec latency 49.866 ms 2024-05-08T20:15:43.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 620920 9.12 MB/sec execute 247 sec latency 12.743 ms 2024-05-08T20:15:43.567 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1084160 16.10 MB/sec execute 247 sec latency 50.314 ms 2024-05-08T20:15:44.194 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:43 smithi083 ceph-mon[38695]: pgmap v489: 129 pgs: 129 active+clean; 194 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.5 MiB/s wr, 177 op/s 2024-05-08T20:15:44.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 622386 9.11 MB/sec execute 248 sec latency 11.061 ms 2024-05-08T20:15:44.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:43 smithi188 ceph-mon[48761]: pgmap v489: 129 pgs: 129 active+clean; 194 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.5 MiB/s wr, 177 op/s 2024-05-08T20:15:44.567 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1087649 16.11 MB/sec execute 248 sec latency 58.428 ms 2024-05-08T20:15:45.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 624032 9.10 MB/sec execute 249 sec latency 22.081 ms 2024-05-08T20:15:45.567 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1091192 16.12 MB/sec execute 249 sec latency 58.030 ms 2024-05-08T20:15:46.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:45 smithi083 ceph-mon[38695]: pgmap v490: 129 pgs: 129 active+clean; 191 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.0 MiB/s wr, 192 op/s 2024-05-08T20:15:46.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 625669 9.11 MB/sec execute 250 sec latency 10.829 ms 2024-05-08T20:15:46.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:45 smithi188 ceph-mon[48761]: pgmap v490: 129 pgs: 129 active+clean; 191 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.0 MiB/s wr, 192 op/s 2024-05-08T20:15:46.567 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1093920 16.12 MB/sec execute 250 sec latency 46.200 ms 2024-05-08T20:15:47.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 627385 9.10 MB/sec execute 251 sec latency 7.891 ms 2024-05-08T20:15:47.567 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1097355 16.13 MB/sec execute 251 sec latency 58.419 ms 2024-05-08T20:15:48.083 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:47 smithi083 ceph-mon[38695]: pgmap v491: 129 pgs: 129 active+clean; 190 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 181 op/s 2024-05-08T20:15:48.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 629004 9.11 MB/sec execute 252 sec latency 14.009 ms 2024-05-08T20:15:48.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:47 smithi188 ceph-mon[48761]: pgmap v491: 129 pgs: 129 active+clean; 190 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 181 op/s 2024-05-08T20:15:48.567 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1100581 16.13 MB/sec execute 252 sec latency 58.397 ms 2024-05-08T20:15:49.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 630528 9.10 MB/sec execute 253 sec latency 10.078 ms 2024-05-08T20:15:49.567 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1103490 16.12 MB/sec execute 253 sec latency 54.530 ms 2024-05-08T20:15:50.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:49 smithi083 ceph-mon[38695]: pgmap v492: 129 pgs: 129 active+clean; 190 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.8 MiB/s wr, 191 op/s 2024-05-08T20:15:50.237 INFO:tasks.workunit.client.1.smithi188.stdout: 1 632375 9.12 MB/sec execute 254 sec latency 14.369 ms 2024-05-08T20:15:50.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:49 smithi188 ceph-mon[48761]: pgmap v492: 129 pgs: 129 active+clean; 190 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.8 MiB/s wr, 191 op/s 2024-05-08T20:15:50.567 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1105620 16.11 MB/sec execute 254 sec latency 95.870 ms 2024-05-08T20:15:51.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 633777 9.10 MB/sec execute 255 sec latency 11.013 ms 2024-05-08T20:15:51.568 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1109042 16.12 MB/sec execute 255 sec latency 54.018 ms 2024-05-08T20:15:52.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:51 smithi083 ceph-mon[38695]: pgmap v493: 129 pgs: 129 active+clean; 191 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.3 MiB/s wr, 210 op/s 2024-05-08T20:15:52.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 635504 9.10 MB/sec execute 256 sec latency 8.973 ms 2024-05-08T20:15:52.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:51 smithi188 ceph-mon[48761]: pgmap v493: 129 pgs: 129 active+clean; 191 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.3 MiB/s wr, 210 op/s 2024-05-08T20:15:52.567 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1112524 16.13 MB/sec execute 256 sec latency 58.482 ms 2024-05-08T20:15:53.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 636922 9.09 MB/sec execute 257 sec latency 4.824 ms 2024-05-08T20:15:53.567 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1116008 16.14 MB/sec execute 257 sec latency 74.751 ms 2024-05-08T20:15:54.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:53 smithi083 ceph-mon[38695]: pgmap v494: 129 pgs: 129 active+clean; 191 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.6 MiB/s wr, 189 op/s 2024-05-08T20:15:54.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:53 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:15:54.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 638721 9.10 MB/sec execute 258 sec latency 21.882 ms 2024-05-08T20:15:54.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:53 smithi188 ceph-mon[48761]: pgmap v494: 129 pgs: 129 active+clean; 191 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.6 MiB/s wr, 189 op/s 2024-05-08T20:15:54.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:53 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:15:54.568 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1118629 16.13 MB/sec execute 258 sec latency 59.738 ms 2024-05-08T20:15:55.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 640130 9.09 MB/sec execute 259 sec latency 5.136 ms 2024-05-08T20:15:55.568 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1119863 16.09 MB/sec execute 259 sec latency 273.210 ms 2024-05-08T20:15:56.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:55 smithi083 ceph-mon[38695]: pgmap v495: 129 pgs: 129 active+clean; 193 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.0 MiB/s wr, 187 op/s 2024-05-08T20:15:56.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 641759 9.08 MB/sec execute 260 sec latency 27.024 ms 2024-05-08T20:15:56.341 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:55 smithi188 ceph-mon[48761]: pgmap v495: 129 pgs: 129 active+clean; 193 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.0 MiB/s wr, 187 op/s 2024-05-08T20:15:56.568 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1121221 16.06 MB/sec execute 260 sec latency 366.728 ms 2024-05-08T20:15:57.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 643370 9.09 MB/sec execute 261 sec latency 5.578 ms 2024-05-08T20:15:57.568 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1123905 16.06 MB/sec execute 261 sec latency 57.760 ms 2024-05-08T20:15:58.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:57 smithi083 ceph-mon[38695]: pgmap v496: 129 pgs: 129 active+clean; 194 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.1 MiB/s wr, 189 op/s 2024-05-08T20:15:58.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 645066 9.08 MB/sec execute 262 sec latency 20.997 ms 2024-05-08T20:15:58.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:57 smithi188 ceph-mon[48761]: pgmap v496: 129 pgs: 129 active+clean; 194 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.1 MiB/s wr, 189 op/s 2024-05-08T20:15:58.568 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1126779 16.05 MB/sec execute 262 sec latency 58.476 ms 2024-05-08T20:15:59.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 646717 9.10 MB/sec execute 263 sec latency 9.687 ms 2024-05-08T20:15:59.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1130246 16.07 MB/sec execute 263 sec latency 60.541 ms 2024-05-08T20:16:00.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:59 smithi083 ceph-mon[38695]: pgmap v497: 129 pgs: 129 active+clean; 191 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.1 MiB/s wr, 209 op/s 2024-05-08T20:16:00.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:16:00.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:16:00.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:16:00.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:15:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:16:00.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 648208 9.08 MB/sec execute 264 sec latency 15.847 ms 2024-05-08T20:16:00.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:59 smithi188 ceph-mon[48761]: pgmap v497: 129 pgs: 129 active+clean; 191 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.1 MiB/s wr, 209 op/s 2024-05-08T20:16:00.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:16:00.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:16:00.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:16:00.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:15:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:16:00.568 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1133789 16.08 MB/sec execute 264 sec latency 60.611 ms 2024-05-08T20:16:01.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 650023 9.09 MB/sec execute 265 sec latency 4.972 ms 2024-05-08T20:16:01.568 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1136404 16.07 MB/sec execute 265 sec latency 60.059 ms 2024-05-08T20:16:02.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:01 smithi083 ceph-mon[38695]: pgmap v498: 129 pgs: 129 active+clean; 192 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.5 MiB/s wr, 218 op/s 2024-05-08T20:16:02.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 651473 9.09 MB/sec execute 266 sec latency 12.190 ms 2024-05-08T20:16:02.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:01 smithi188 ceph-mon[48761]: pgmap v498: 129 pgs: 129 active+clean; 192 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.5 MiB/s wr, 218 op/s 2024-05-08T20:16:02.568 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1139031 16.05 MB/sec execute 266 sec latency 61.820 ms 2024-05-08T20:16:03.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 653232 9.09 MB/sec execute 267 sec latency 10.044 ms 2024-05-08T20:16:03.568 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1141689 16.06 MB/sec execute 267 sec latency 58.299 ms 2024-05-08T20:16:04.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:03 smithi083 ceph-mon[38695]: pgmap v499: 129 pgs: 129 active+clean; 192 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.3 MiB/s wr, 158 op/s 2024-05-08T20:16:04.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 654678 9.08 MB/sec execute 268 sec latency 6.167 ms 2024-05-08T20:16:04.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:03 smithi188 ceph-mon[48761]: pgmap v499: 129 pgs: 129 active+clean; 192 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.3 MiB/s wr, 158 op/s 2024-05-08T20:16:04.568 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1144630 16.05 MB/sec execute 268 sec latency 53.003 ms 2024-05-08T20:16:05.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 656648 9.09 MB/sec execute 269 sec latency 13.524 ms 2024-05-08T20:16:05.568 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1148093 16.06 MB/sec execute 269 sec latency 57.954 ms 2024-05-08T20:16:06.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:05 smithi083 ceph-mon[38695]: pgmap v500: 129 pgs: 129 active+clean; 187 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 3.8 MiB/s wr, 175 op/s 2024-05-08T20:16:06.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 658039 9.08 MB/sec execute 270 sec latency 15.290 ms 2024-05-08T20:16:06.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:05 smithi188 ceph-mon[48761]: pgmap v500: 129 pgs: 129 active+clean; 187 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 3.8 MiB/s wr, 175 op/s 2024-05-08T20:16:06.568 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1151570 16.07 MB/sec execute 270 sec latency 55.803 ms 2024-05-08T20:16:07.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 659897 9.09 MB/sec execute 271 sec latency 16.107 ms 2024-05-08T20:16:07.568 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1154130 16.07 MB/sec execute 271 sec latency 141.684 ms 2024-05-08T20:16:08.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:07 smithi083 ceph-mon[38695]: pgmap v501: 129 pgs: 129 active+clean; 191 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.1 MiB/s wr, 193 op/s 2024-05-08T20:16:08.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 661393 9.08 MB/sec execute 272 sec latency 10.555 ms 2024-05-08T20:16:08.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:07 smithi188 ceph-mon[48761]: pgmap v501: 129 pgs: 129 active+clean; 191 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.1 MiB/s wr, 193 op/s 2024-05-08T20:16:08.568 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1157150 16.07 MB/sec execute 272 sec latency 56.458 ms 2024-05-08T20:16:09.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:16:09.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 663032 9.07 MB/sec execute 273 sec latency 12.860 ms 2024-05-08T20:16:09.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:16:09.568 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1159820 16.06 MB/sec execute 273 sec latency 56.190 ms 2024-05-08T20:16:10.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:09 smithi083 ceph-mon[38695]: pgmap v502: 129 pgs: 129 active+clean; 194 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.1 MiB/s wr, 213 op/s 2024-05-08T20:16:10.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 664673 9.08 MB/sec execute 274 sec latency 7.020 ms 2024-05-08T20:16:10.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:09 smithi188 ceph-mon[48761]: pgmap v502: 129 pgs: 129 active+clean; 194 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.1 MiB/s wr, 213 op/s 2024-05-08T20:16:10.568 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1162401 16.05 MB/sec execute 274 sec latency 55.684 ms 2024-05-08T20:16:11.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 666414 9.07 MB/sec execute 275 sec latency 9.615 ms 2024-05-08T20:16:11.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1165825 16.06 MB/sec execute 275 sec latency 55.618 ms 2024-05-08T20:16:12.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:11 smithi083 ceph-mon[38695]: pgmap v503: 129 pgs: 129 active+clean; 195 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.2 MiB/s wr, 188 op/s 2024-05-08T20:16:12.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 668069 9.08 MB/sec execute 276 sec latency 6.612 ms 2024-05-08T20:16:12.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:11 smithi188 ceph-mon[48761]: pgmap v503: 129 pgs: 129 active+clean; 195 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.2 MiB/s wr, 188 op/s 2024-05-08T20:16:12.568 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1169312 16.07 MB/sec execute 276 sec latency 57.282 ms 2024-05-08T20:16:13.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 669719 9.07 MB/sec execute 277 sec latency 25.398 ms 2024-05-08T20:16:13.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1172797 16.08 MB/sec execute 277 sec latency 68.068 ms 2024-05-08T20:16:14.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:13 smithi083 ceph-mon[38695]: pgmap v504: 129 pgs: 129 active+clean; 195 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.3 MiB/s wr, 151 op/s 2024-05-08T20:16:14.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 671483 9.09 MB/sec execute 278 sec latency 13.295 ms 2024-05-08T20:16:14.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:13 smithi188 ceph-mon[48761]: pgmap v504: 129 pgs: 129 active+clean; 195 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.3 MiB/s wr, 151 op/s 2024-05-08T20:16:14.568 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1175436 16.07 MB/sec execute 278 sec latency 59.589 ms 2024-05-08T20:16:15.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 673034 9.07 MB/sec execute 279 sec latency 15.905 ms 2024-05-08T20:16:15.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1178423 16.07 MB/sec execute 279 sec latency 68.684 ms 2024-05-08T20:16:16.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:15 smithi083 ceph-mon[38695]: pgmap v505: 129 pgs: 129 active+clean; 194 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.1 MiB/s wr, 206 op/s 2024-05-08T20:16:16.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 674904 9.09 MB/sec execute 280 sec latency 7.967 ms 2024-05-08T20:16:16.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:15 smithi188 ceph-mon[48761]: pgmap v505: 129 pgs: 129 active+clean; 194 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.1 MiB/s wr, 206 op/s 2024-05-08T20:16:16.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1181095 16.06 MB/sec execute 280 sec latency 64.804 ms 2024-05-08T20:16:17.238 INFO:tasks.workunit.client.1.smithi188.stdout: 1 676378 9.08 MB/sec execute 281 sec latency 9.148 ms 2024-05-08T20:16:17.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1183741 16.06 MB/sec execute 281 sec latency 57.635 ms 2024-05-08T20:16:18.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:17 smithi083 ceph-mon[38695]: pgmap v506: 129 pgs: 129 active+clean; 192 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.3 MiB/s wr, 208 op/s 2024-05-08T20:16:18.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 678131 9.08 MB/sec execute 282 sec latency 4.769 ms 2024-05-08T20:16:18.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:17 smithi188 ceph-mon[48761]: pgmap v506: 129 pgs: 129 active+clean; 192 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.3 MiB/s wr, 208 op/s 2024-05-08T20:16:18.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1187100 16.07 MB/sec execute 282 sec latency 66.745 ms 2024-05-08T20:16:19.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 679693 9.08 MB/sec execute 283 sec latency 5.028 ms 2024-05-08T20:16:19.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1190581 16.08 MB/sec execute 283 sec latency 57.672 ms 2024-05-08T20:16:20.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:19 smithi083 ceph-mon[38695]: pgmap v507: 129 pgs: 129 active+clean; 196 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.2 MiB/s wr, 204 op/s 2024-05-08T20:16:20.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 681588 9.08 MB/sec execute 284 sec latency 15.594 ms 2024-05-08T20:16:20.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:19 smithi188 ceph-mon[48761]: pgmap v507: 129 pgs: 129 active+clean; 196 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.2 MiB/s wr, 204 op/s 2024-05-08T20:16:20.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1194067 16.09 MB/sec execute 284 sec latency 66.748 ms 2024-05-08T20:16:21.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 682916 9.07 MB/sec execute 285 sec latency 14.229 ms 2024-05-08T20:16:21.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1196708 16.08 MB/sec execute 285 sec latency 58.367 ms 2024-05-08T20:16:22.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:21 smithi083 ceph-mon[38695]: pgmap v508: 129 pgs: 129 active+clean; 197 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.4 MiB/s wr, 199 op/s 2024-05-08T20:16:22.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 684731 9.07 MB/sec execute 286 sec latency 27.393 ms 2024-05-08T20:16:22.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:21 smithi188 ceph-mon[48761]: pgmap v508: 129 pgs: 129 active+clean; 197 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.4 MiB/s wr, 199 op/s 2024-05-08T20:16:22.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1199347 16.07 MB/sec execute 286 sec latency 130.936 ms 2024-05-08T20:16:23.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 686186 9.07 MB/sec execute 287 sec latency 6.219 ms 2024-05-08T20:16:23.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1202364 16.07 MB/sec execute 287 sec latency 46.952 ms 2024-05-08T20:16:24.092 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:23 smithi188 ceph-mon[48761]: pgmap v509: 129 pgs: 129 active+clean; 197 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.5 MiB/s wr, 177 op/s 2024-05-08T20:16:24.092 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:16:24.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:23 smithi083 ceph-mon[38695]: pgmap v509: 129 pgs: 129 active+clean; 197 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.5 MiB/s wr, 177 op/s 2024-05-08T20:16:24.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:16:24.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 687833 9.06 MB/sec execute 288 sec latency 15.527 ms 2024-05-08T20:16:24.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1204947 16.07 MB/sec execute 288 sec latency 54.921 ms 2024-05-08T20:16:25.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 689425 9.07 MB/sec execute 289 sec latency 12.736 ms 2024-05-08T20:16:25.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1208371 16.08 MB/sec execute 289 sec latency 54.767 ms 2024-05-08T20:16:26.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:25 smithi083 ceph-mon[38695]: pgmap v510: 129 pgs: 129 active+clean; 192 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.0 MiB/s wr, 198 op/s 2024-05-08T20:16:26.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 691040 9.06 MB/sec execute 290 sec latency 8.222 ms 2024-05-08T20:16:26.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:25 smithi188 ceph-mon[48761]: pgmap v510: 129 pgs: 129 active+clean; 192 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.0 MiB/s wr, 198 op/s 2024-05-08T20:16:26.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1211851 16.09 MB/sec execute 290 sec latency 54.679 ms 2024-05-08T20:16:27.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 692775 9.07 MB/sec execute 291 sec latency 12.459 ms 2024-05-08T20:16:27.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1215340 16.10 MB/sec execute 291 sec latency 105.609 ms 2024-05-08T20:16:28.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:27 smithi083 ceph-mon[38695]: pgmap v511: 129 pgs: 129 active+clean; 196 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.3 MiB/s wr, 196 op/s 2024-05-08T20:16:28.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 694198 9.06 MB/sec execute 292 sec latency 13.996 ms 2024-05-08T20:16:28.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:27 smithi188 ceph-mon[48761]: pgmap v511: 129 pgs: 129 active+clean; 196 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.3 MiB/s wr, 196 op/s 2024-05-08T20:16:28.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1215345 16.04 MB/sec execute 292 sec latency 339.384 ms 2024-05-08T20:16:29.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 696004 9.07 MB/sec execute 293 sec latency 7.681 ms 2024-05-08T20:16:29.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1217080 16.01 MB/sec execute 293 sec latency 404.748 ms 2024-05-08T20:16:30.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:29 smithi083 ceph-mon[38695]: pgmap v512: 129 pgs: 129 active+clean; 198 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.2 MiB/s wr, 212 op/s 2024-05-08T20:16:30.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 697428 9.07 MB/sec execute 294 sec latency 11.517 ms 2024-05-08T20:16:30.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:29 smithi188 ceph-mon[48761]: pgmap v512: 129 pgs: 129 active+clean; 198 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.2 MiB/s wr, 212 op/s 2024-05-08T20:16:30.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1220100 16.01 MB/sec execute 294 sec latency 55.715 ms 2024-05-08T20:16:31.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 699319 9.07 MB/sec execute 295 sec latency 11.345 ms 2024-05-08T20:16:31.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1222746 16.01 MB/sec execute 295 sec latency 53.602 ms 2024-05-08T20:16:32.191 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:31 smithi083 ceph-mon[38695]: pgmap v513: 129 pgs: 129 active+clean; 193 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.1 MiB/s wr, 186 op/s 2024-05-08T20:16:32.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 700557 9.05 MB/sec execute 296 sec latency 16.288 ms 2024-05-08T20:16:32.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:31 smithi188 ceph-mon[48761]: pgmap v513: 129 pgs: 129 active+clean; 193 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.1 MiB/s wr, 186 op/s 2024-05-08T20:16:32.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1226187 16.02 MB/sec execute 296 sec latency 58.490 ms 2024-05-08T20:16:33.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 702331 9.06 MB/sec execute 297 sec latency 7.915 ms 2024-05-08T20:16:33.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1229649 16.03 MB/sec execute 297 sec latency 48.935 ms 2024-05-08T20:16:34.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:33 smithi083 ceph-mon[38695]: pgmap v514: 129 pgs: 129 active+clean; 194 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.3 MiB/s wr, 152 op/s 2024-05-08T20:16:34.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 703825 9.06 MB/sec execute 298 sec latency 15.127 ms 2024-05-08T20:16:34.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:33 smithi188 ceph-mon[48761]: pgmap v514: 129 pgs: 129 active+clean; 194 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.3 MiB/s wr, 152 op/s 2024-05-08T20:16:34.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1233130 16.04 MB/sec execute 298 sec latency 57.791 ms 2024-05-08T20:16:35.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 705526 9.05 MB/sec execute 299 sec latency 20.516 ms 2024-05-08T20:16:35.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1236613 16.05 MB/sec execute 299 sec latency 57.508 ms 2024-05-08T20:16:36.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:35 smithi083 ceph-mon[38695]: pgmap v515: 129 pgs: 129 active+clean; 196 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.0 MiB/s wr, 204 op/s 2024-05-08T20:16:36.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 707139 9.06 MB/sec execute 300 sec latency 10.320 ms 2024-05-08T20:16:36.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:35 smithi188 ceph-mon[48761]: pgmap v515: 129 pgs: 129 active+clean; 196 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.0 MiB/s wr, 204 op/s 2024-05-08T20:16:36.570 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1239129 16.04 MB/sec execute 300 sec latency 63.320 ms 2024-05-08T20:16:37.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 708774 9.05 MB/sec execute 301 sec latency 4.660 ms 2024-05-08T20:16:37.569 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1241900 16.03 MB/sec execute 301 sec latency 50.187 ms 2024-05-08T20:16:38.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:37 smithi083 ceph-mon[38695]: pgmap v516: 129 pgs: 129 active+clean; 192 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.1 MiB/s wr, 200 op/s 2024-05-08T20:16:38.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 710516 9.06 MB/sec execute 302 sec latency 6.100 ms 2024-05-08T20:16:38.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:37 smithi188 ceph-mon[48761]: pgmap v516: 129 pgs: 129 active+clean; 192 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.1 MiB/s wr, 200 op/s 2024-05-08T20:16:38.570 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1244916 16.03 MB/sec execute 302 sec latency 46.556 ms 2024-05-08T20:16:39.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 711989 9.05 MB/sec execute 303 sec latency 14.308 ms 2024-05-08T20:16:39.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:16:39.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:16:39.570 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1247383 16.03 MB/sec execute 303 sec latency 150.660 ms 2024-05-08T20:16:40.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 713771 9.06 MB/sec execute 304 sec latency 10.167 ms 2024-05-08T20:16:40.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:39 smithi188 ceph-mon[48761]: pgmap v517: 129 pgs: 129 active+clean; 195 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.0 MiB/s wr, 197 op/s 2024-05-08T20:16:40.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:39 smithi083 ceph-mon[38695]: pgmap v517: 129 pgs: 129 active+clean; 195 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.0 MiB/s wr, 197 op/s 2024-05-08T20:16:40.570 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1250856 16.04 MB/sec execute 304 sec latency 58.383 ms 2024-05-08T20:16:41.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 715194 9.05 MB/sec execute 305 sec latency 12.362 ms 2024-05-08T20:16:41.570 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1254350 16.05 MB/sec execute 305 sec latency 50.090 ms 2024-05-08T20:16:42.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 717058 9.05 MB/sec execute 306 sec latency 22.450 ms 2024-05-08T20:16:42.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:41 smithi188 ceph-mon[48761]: pgmap v518: 129 pgs: 129 active+clean; 197 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.2 MiB/s wr, 193 op/s 2024-05-08T20:16:42.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:41 smithi083 ceph-mon[38695]: pgmap v518: 129 pgs: 129 active+clean; 197 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.2 MiB/s wr, 193 op/s 2024-05-08T20:16:42.570 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1257000 16.04 MB/sec execute 306 sec latency 56.049 ms 2024-05-08T20:16:43.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 718373 9.04 MB/sec execute 307 sec latency 17.019 ms 2024-05-08T20:16:43.570 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1260127 16.04 MB/sec execute 307 sec latency 58.398 ms 2024-05-08T20:16:44.239 INFO:tasks.workunit.client.1.smithi188.stdout: 1 720090 9.04 MB/sec execute 308 sec latency 12.455 ms 2024-05-08T20:16:44.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:43 smithi188 ceph-mon[48761]: pgmap v519: 129 pgs: 129 active+clean; 197 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.5 MiB/s wr, 168 op/s 2024-05-08T20:16:44.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:43 smithi083 ceph-mon[38695]: pgmap v519: 129 pgs: 129 active+clean; 197 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.5 MiB/s wr, 168 op/s 2024-05-08T20:16:44.570 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1262647 16.03 MB/sec execute 308 sec latency 60.980 ms 2024-05-08T20:16:45.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 721580 9.04 MB/sec execute 309 sec latency 6.862 ms 2024-05-08T20:16:45.570 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1265288 16.03 MB/sec execute 309 sec latency 52.956 ms 2024-05-08T20:16:46.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 723234 9.04 MB/sec execute 310 sec latency 22.548 ms 2024-05-08T20:16:46.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:45 smithi188 ceph-mon[48761]: pgmap v520: 129 pgs: 129 active+clean; 198 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.0 MiB/s wr, 189 op/s 2024-05-08T20:16:46.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:45 smithi083 ceph-mon[38695]: pgmap v520: 129 pgs: 129 active+clean; 198 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.0 MiB/s wr, 189 op/s 2024-05-08T20:16:46.570 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1268658 16.04 MB/sec execute 310 sec latency 65.383 ms 2024-05-08T20:16:47.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 724844 9.05 MB/sec execute 311 sec latency 5.910 ms 2024-05-08T20:16:47.570 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1272133 16.05 MB/sec execute 311 sec latency 52.514 ms 2024-05-08T20:16:48.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 726464 9.03 MB/sec execute 312 sec latency 8.550 ms 2024-05-08T20:16:48.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:47 smithi188 ceph-mon[48761]: pgmap v521: 129 pgs: 129 active+clean; 196 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.2 MiB/s wr, 195 op/s 2024-05-08T20:16:48.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:47 smithi083 ceph-mon[38695]: pgmap v521: 129 pgs: 129 active+clean; 196 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.2 MiB/s wr, 195 op/s 2024-05-08T20:16:48.570 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1275626 16.05 MB/sec execute 312 sec latency 59.530 ms 2024-05-08T20:16:49.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 728209 9.05 MB/sec execute 313 sec latency 6.245 ms 2024-05-08T20:16:49.570 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1279156 16.06 MB/sec execute 313 sec latency 61.444 ms 2024-05-08T20:16:50.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 729639 9.04 MB/sec execute 314 sec latency 16.528 ms 2024-05-08T20:16:50.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:49 smithi188 ceph-mon[48761]: pgmap v522: 129 pgs: 129 active+clean; 196 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.1 MiB/s wr, 192 op/s 2024-05-08T20:16:50.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:49 smithi083 ceph-mon[38695]: pgmap v522: 129 pgs: 129 active+clean; 196 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.1 MiB/s wr, 192 op/s 2024-05-08T20:16:50.570 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1281477 16.05 MB/sec execute 314 sec latency 63.852 ms 2024-05-08T20:16:51.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 731461 9.05 MB/sec execute 315 sec latency 13.832 ms 2024-05-08T20:16:51.570 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1284445 16.04 MB/sec execute 315 sec latency 55.679 ms 2024-05-08T20:16:52.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 732902 9.04 MB/sec execute 316 sec latency 5.956 ms 2024-05-08T20:16:52.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:51 smithi188 ceph-mon[48761]: pgmap v523: 129 pgs: 129 active+clean; 195 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 176 op/s 2024-05-08T20:16:52.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:51 smithi083 ceph-mon[38695]: pgmap v523: 129 pgs: 129 active+clean; 195 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 176 op/s 2024-05-08T20:16:52.570 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1287466 16.05 MB/sec execute 316 sec latency 58.476 ms 2024-05-08T20:16:53.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 734784 9.04 MB/sec execute 317 sec latency 8.810 ms 2024-05-08T20:16:53.570 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1291009 16.06 MB/sec execute 317 sec latency 74.349 ms 2024-05-08T20:16:54.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 736105 9.03 MB/sec execute 318 sec latency 18.418 ms 2024-05-08T20:16:54.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:53 smithi188 ceph-mon[48761]: pgmap v524: 129 pgs: 129 active+clean; 194 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.4 MiB/s wr, 183 op/s 2024-05-08T20:16:54.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:53 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:16:54.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:53 smithi083 ceph-mon[38695]: pgmap v524: 129 pgs: 129 active+clean; 194 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.4 MiB/s wr, 183 op/s 2024-05-08T20:16:54.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:53 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:16:54.570 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1294150 16.07 MB/sec execute 318 sec latency 49.262 ms 2024-05-08T20:16:55.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 737973 9.04 MB/sec execute 319 sec latency 21.479 ms 2024-05-08T20:16:55.570 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1297058 16.06 MB/sec execute 319 sec latency 91.776 ms 2024-05-08T20:16:56.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 739470 9.03 MB/sec execute 320 sec latency 13.155 ms 2024-05-08T20:16:56.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:55 smithi188 ceph-mon[48761]: pgmap v525: 129 pgs: 129 active+clean; 195 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 3.9 MiB/s wr, 198 op/s 2024-05-08T20:16:56.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:55 smithi083 ceph-mon[38695]: pgmap v525: 129 pgs: 129 active+clean; 195 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 3.9 MiB/s wr, 198 op/s 2024-05-08T20:16:56.570 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1300537 16.07 MB/sec execute 320 sec latency 58.503 ms 2024-05-08T20:16:57.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 741166 9.03 MB/sec execute 321 sec latency 23.336 ms 2024-05-08T20:16:57.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1304099 16.08 MB/sec execute 321 sec latency 48.067 ms 2024-05-08T20:16:58.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 742796 9.03 MB/sec execute 322 sec latency 12.821 ms 2024-05-08T20:16:58.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:57 smithi188 ceph-mon[48761]: pgmap v526: 129 pgs: 129 active+clean; 195 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.0 MiB/s wr, 204 op/s 2024-05-08T20:16:58.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:57 smithi083 ceph-mon[38695]: pgmap v526: 129 pgs: 129 active+clean; 195 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.0 MiB/s wr, 204 op/s 2024-05-08T20:16:58.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1307591 16.09 MB/sec execute 322 sec latency 58.441 ms 2024-05-08T20:16:59.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 744522 9.03 MB/sec execute 323 sec latency 17.372 ms 2024-05-08T20:16:59.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1310243 16.08 MB/sec execute 323 sec latency 52.016 ms 2024-05-08T20:17:00.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 746118 9.04 MB/sec execute 324 sec latency 5.954 ms 2024-05-08T20:17:00.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:59 smithi188 ceph-mon[48761]: pgmap v527: 129 pgs: 129 active+clean; 192 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 3.8 MiB/s wr, 171 op/s 2024-05-08T20:17:00.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:16:59 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:17:00.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:59 smithi083 ceph-mon[38695]: pgmap v527: 129 pgs: 129 active+clean; 192 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 3.8 MiB/s wr, 171 op/s 2024-05-08T20:17:00.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:16:59 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:17:00.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1311194 16.05 MB/sec execute 324 sec latency 557.213 ms 2024-05-08T20:17:01.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 747739 9.03 MB/sec execute 325 sec latency 8.903 ms 2024-05-08T20:17:01.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:00 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:17:01.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:00 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:17:01.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:00 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:17:01.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:00 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:17:01.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:00 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:17:01.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:00 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:17:01.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:00 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:17:01.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:00 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:17:01.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:00 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:17:01.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:00 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:17:01.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:00 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:17:01.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:00 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-08T20:17:01.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:00 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:17:01.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:00 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:17:01.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:00 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:17:01.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:00 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:17:01.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1313453 16.04 MB/sec execute 325 sec latency 301.815 ms 2024-05-08T20:17:02.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 749523 9.04 MB/sec execute 326 sec latency 4.202 ms 2024-05-08T20:17:02.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:01 smithi188 ceph-mon[48761]: pgmap v528: 129 pgs: 129 active+clean; 197 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.3 MiB/s wr, 219 op/s 2024-05-08T20:17:02.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:01 smithi188 ceph-mon[48761]: Adjusting osd_memory_target on smithi188 to 1688M 2024-05-08T20:17:02.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:01 smithi083 ceph-mon[38695]: pgmap v528: 129 pgs: 129 active+clean; 197 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.3 MiB/s wr, 219 op/s 2024-05-08T20:17:02.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:01 smithi083 ceph-mon[38695]: Adjusting osd_memory_target on smithi188 to 1688M 2024-05-08T20:17:02.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1316375 16.03 MB/sec execute 326 sec latency 60.248 ms 2024-05-08T20:17:03.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 751048 9.03 MB/sec execute 327 sec latency 11.872 ms 2024-05-08T20:17:03.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1319187 16.04 MB/sec execute 327 sec latency 60.121 ms 2024-05-08T20:17:04.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 752859 9.04 MB/sec execute 328 sec latency 6.353 ms 2024-05-08T20:17:04.250 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:03 smithi188 ceph-mon[48761]: pgmap v529: 129 pgs: 129 active+clean; 198 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.8 MiB/s wr, 186 op/s 2024-05-08T20:17:04.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:03 smithi083 ceph-mon[38695]: pgmap v529: 129 pgs: 129 active+clean; 198 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.8 MiB/s wr, 186 op/s 2024-05-08T20:17:04.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1322078 16.03 MB/sec execute 328 sec latency 58.418 ms 2024-05-08T20:17:05.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 754284 9.03 MB/sec execute 329 sec latency 9.194 ms 2024-05-08T20:17:05.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1325446 16.04 MB/sec execute 329 sec latency 60.147 ms 2024-05-08T20:17:06.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 756097 9.03 MB/sec execute 330 sec latency 10.352 ms 2024-05-08T20:17:06.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:05 smithi188 ceph-mon[48761]: pgmap v530: 129 pgs: 129 active+clean; 198 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.0 MiB/s wr, 187 op/s 2024-05-08T20:17:06.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:05 smithi083 ceph-mon[38695]: pgmap v530: 129 pgs: 129 active+clean; 198 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.0 MiB/s wr, 187 op/s 2024-05-08T20:17:06.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1328919 16.05 MB/sec execute 330 sec latency 60.266 ms 2024-05-08T20:17:07.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 757501 9.02 MB/sec execute 331 sec latency 7.754 ms 2024-05-08T20:17:07.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1332410 16.06 MB/sec execute 331 sec latency 68.392 ms 2024-05-08T20:17:08.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 759335 9.03 MB/sec execute 332 sec latency 14.056 ms 2024-05-08T20:17:08.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:07 smithi188 ceph-mon[48761]: pgmap v531: 129 pgs: 129 active+clean; 196 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.2 MiB/s wr, 191 op/s 2024-05-08T20:17:08.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:07 smithi083 ceph-mon[38695]: pgmap v531: 129 pgs: 129 active+clean; 196 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.2 MiB/s wr, 191 op/s 2024-05-08T20:17:08.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1335031 16.05 MB/sec execute 332 sec latency 49.624 ms 2024-05-08T20:17:09.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:17:09.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 760762 9.02 MB/sec execute 333 sec latency 14.193 ms 2024-05-08T20:17:09.312 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:17:09.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1338190 16.06 MB/sec execute 333 sec latency 51.889 ms 2024-05-08T20:17:10.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 762417 9.02 MB/sec execute 334 sec latency 27.550 ms 2024-05-08T20:17:10.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:09 smithi188 ceph-mon[48761]: pgmap v532: 129 pgs: 129 active+clean; 197 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.2 MiB/s wr, 206 op/s 2024-05-08T20:17:10.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:09 smithi083 ceph-mon[38695]: pgmap v532: 129 pgs: 129 active+clean; 197 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.2 MiB/s wr, 206 op/s 2024-05-08T20:17:10.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1340713 16.04 MB/sec execute 334 sec latency 57.537 ms 2024-05-08T20:17:11.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 764003 9.03 MB/sec execute 335 sec latency 8.927 ms 2024-05-08T20:17:11.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1343248 16.04 MB/sec execute 335 sec latency 94.165 ms 2024-05-08T20:17:12.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 765669 9.02 MB/sec execute 336 sec latency 14.517 ms 2024-05-08T20:17:12.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:11 smithi188 ceph-mon[48761]: pgmap v533: 129 pgs: 129 active+clean; 193 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.4 MiB/s wr, 228 op/s 2024-05-08T20:17:12.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:11 smithi083 ceph-mon[38695]: pgmap v533: 129 pgs: 129 active+clean; 193 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.4 MiB/s wr, 228 op/s 2024-05-08T20:17:12.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1346716 16.05 MB/sec execute 336 sec latency 48.736 ms 2024-05-08T20:17:13.240 INFO:tasks.workunit.client.1.smithi188.stdout: 1 767312 9.03 MB/sec execute 337 sec latency 8.995 ms 2024-05-08T20:17:13.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1350188 16.06 MB/sec execute 337 sec latency 60.142 ms 2024-05-08T20:17:14.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:13 smithi083 ceph-mon[38695]: pgmap v534: 129 pgs: 129 active+clean; 193 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.3 MiB/s wr, 165 op/s 2024-05-08T20:17:14.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 768804 9.02 MB/sec execute 338 sec latency 12.265 ms 2024-05-08T20:17:14.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:13 smithi188 ceph-mon[48761]: pgmap v534: 129 pgs: 129 active+clean; 193 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.3 MiB/s wr, 165 op/s 2024-05-08T20:17:14.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1353679 16.06 MB/sec execute 338 sec latency 60.118 ms 2024-05-08T20:17:15.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 770608 9.03 MB/sec execute 339 sec latency 15.211 ms 2024-05-08T20:17:15.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1355999 16.05 MB/sec execute 339 sec latency 66.707 ms 2024-05-08T20:17:16.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 772005 9.02 MB/sec execute 340 sec latency 12.874 ms 2024-05-08T20:17:16.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:15 smithi188 ceph-mon[48761]: pgmap v535: 129 pgs: 129 active+clean; 194 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.8 MiB/s wr, 181 op/s 2024-05-08T20:17:16.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:15 smithi083 ceph-mon[38695]: pgmap v535: 129 pgs: 129 active+clean; 194 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.8 MiB/s wr, 181 op/s 2024-05-08T20:17:16.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1358921 16.05 MB/sec execute 340 sec latency 59.945 ms 2024-05-08T20:17:17.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 773813 9.02 MB/sec execute 341 sec latency 9.997 ms 2024-05-08T20:17:17.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1361730 16.05 MB/sec execute 341 sec latency 58.207 ms 2024-05-08T20:17:18.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 775157 9.01 MB/sec execute 342 sec latency 20.366 ms 2024-05-08T20:17:18.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:17 smithi188 ceph-mon[48761]: pgmap v536: 129 pgs: 129 active+clean; 194 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.2 MiB/s wr, 189 op/s 2024-05-08T20:17:18.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:17 smithi083 ceph-mon[38695]: pgmap v536: 129 pgs: 129 active+clean; 194 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.2 MiB/s wr, 189 op/s 2024-05-08T20:17:18.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1364624 16.05 MB/sec execute 342 sec latency 54.140 ms 2024-05-08T20:17:19.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 776946 9.02 MB/sec execute 343 sec latency 12.680 ms 2024-05-08T20:17:19.571 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1368110 16.05 MB/sec execute 343 sec latency 53.237 ms 2024-05-08T20:17:20.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 778325 9.01 MB/sec execute 344 sec latency 6.644 ms 2024-05-08T20:17:20.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:19 smithi188 ceph-mon[48761]: pgmap v537: 129 pgs: 129 active+clean; 196 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 4.0 MiB/s wr, 203 op/s 2024-05-08T20:17:20.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:19 smithi083 ceph-mon[38695]: pgmap v537: 129 pgs: 129 active+clean; 196 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 4.0 MiB/s wr, 203 op/s 2024-05-08T20:17:20.572 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1371535 16.06 MB/sec execute 344 sec latency 62.441 ms 2024-05-08T20:17:21.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 779963 9.01 MB/sec execute 345 sec latency 21.840 ms 2024-05-08T20:17:21.572 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1375753 16.13 MB/sec execute 345 sec latency 58.728 ms 2024-05-08T20:17:22.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 781594 9.01 MB/sec execute 346 sec latency 6.744 ms 2024-05-08T20:17:22.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:21 smithi188 ceph-mon[48761]: pgmap v538: 129 pgs: 129 active+clean; 190 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.2 MiB/s wr, 182 op/s 2024-05-08T20:17:22.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:21 smithi083 ceph-mon[38695]: pgmap v538: 129 pgs: 129 active+clean; 190 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.2 MiB/s wr, 182 op/s 2024-05-08T20:17:22.572 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1379268 16.14 MB/sec execute 346 sec latency 58.353 ms 2024-05-08T20:17:23.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 783223 9.00 MB/sec execute 347 sec latency 7.395 ms 2024-05-08T20:17:23.572 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1381784 16.13 MB/sec execute 347 sec latency 54.788 ms 2024-05-08T20:17:24.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 784976 9.02 MB/sec execute 348 sec latency 9.528 ms 2024-05-08T20:17:24.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:23 smithi188 ceph-mon[48761]: pgmap v539: 129 pgs: 129 active+clean; 192 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.5 MiB/s wr, 169 op/s 2024-05-08T20:17:24.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:17:24.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:23 smithi083 ceph-mon[38695]: pgmap v539: 129 pgs: 129 active+clean; 192 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.5 MiB/s wr, 169 op/s 2024-05-08T20:17:24.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:17:24.572 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1385113 16.14 MB/sec execute 348 sec latency 54.489 ms 2024-05-08T20:17:25.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 786493 9.01 MB/sec execute 349 sec latency 11.467 ms 2024-05-08T20:17:25.572 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1388451 16.14 MB/sec execute 349 sec latency 58.525 ms 2024-05-08T20:17:26.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 788299 9.02 MB/sec execute 350 sec latency 5.308 ms 2024-05-08T20:17:26.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:25 smithi188 ceph-mon[48761]: pgmap v540: 129 pgs: 129 active+clean; 193 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.0 MiB/s wr, 184 op/s 2024-05-08T20:17:26.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:25 smithi083 ceph-mon[38695]: pgmap v540: 129 pgs: 129 active+clean; 193 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.0 MiB/s wr, 184 op/s 2024-05-08T20:17:26.572 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1391644 16.14 MB/sec execute 350 sec latency 48.717 ms 2024-05-08T20:17:27.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 789724 9.01 MB/sec execute 351 sec latency 11.963 ms 2024-05-08T20:17:27.572 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1395193 16.15 MB/sec execute 351 sec latency 48.420 ms 2024-05-08T20:17:28.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 791544 9.01 MB/sec execute 352 sec latency 13.465 ms 2024-05-08T20:17:28.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:27 smithi188 ceph-mon[48761]: pgmap v541: 129 pgs: 129 active+clean; 193 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.2 MiB/s wr, 214 op/s 2024-05-08T20:17:28.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:27 smithi083 ceph-mon[38695]: pgmap v541: 129 pgs: 129 active+clean; 193 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.2 MiB/s wr, 214 op/s 2024-05-08T20:17:28.572 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1398248 16.15 MB/sec execute 352 sec latency 58.460 ms 2024-05-08T20:17:29.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 792954 9.00 MB/sec execute 353 sec latency 6.250 ms 2024-05-08T20:17:29.572 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1400833 16.15 MB/sec execute 353 sec latency 91.752 ms 2024-05-08T20:17:30.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 794938 9.01 MB/sec execute 354 sec latency 12.102 ms 2024-05-08T20:17:30.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:30 smithi188 ceph-mon[48761]: pgmap v542: 129 pgs: 129 active+clean; 194 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.7 MiB/s wr, 181 op/s 2024-05-08T20:17:30.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:30 smithi083 ceph-mon[38695]: pgmap v542: 129 pgs: 129 active+clean; 194 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.7 MiB/s wr, 181 op/s 2024-05-08T20:17:30.572 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1404258 16.16 MB/sec execute 354 sec latency 55.018 ms 2024-05-08T20:17:31.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 796299 9.00 MB/sec execute 355 sec latency 14.250 ms 2024-05-08T20:17:31.572 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1407738 16.16 MB/sec execute 355 sec latency 98.315 ms 2024-05-08T20:17:32.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 798079 9.01 MB/sec execute 356 sec latency 22.270 ms 2024-05-08T20:17:32.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:32 smithi188 ceph-mon[48761]: pgmap v543: 129 pgs: 129 active+clean; 197 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.1 MiB/s wr, 194 op/s 2024-05-08T20:17:32.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:32 smithi083 ceph-mon[38695]: pgmap v543: 129 pgs: 129 active+clean; 197 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.1 MiB/s wr, 194 op/s 2024-05-08T20:17:32.572 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1407797 16.12 MB/sec execute 356 sec latency 563.591 ms 2024-05-08T20:17:33.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 799610 9.01 MB/sec execute 357 sec latency 6.162 ms 2024-05-08T20:17:33.572 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1411216 16.13 MB/sec execute 357 sec latency 316.172 ms 2024-05-08T20:17:34.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 801296 9.00 MB/sec execute 358 sec latency 19.855 ms 2024-05-08T20:17:34.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:34 smithi188 ceph-mon[48761]: pgmap v544: 129 pgs: 129 active+clean; 192 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.3 MiB/s wr, 172 op/s 2024-05-08T20:17:34.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:34 smithi083 ceph-mon[38695]: pgmap v544: 129 pgs: 129 active+clean; 192 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.3 MiB/s wr, 172 op/s 2024-05-08T20:17:34.572 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1413733 16.12 MB/sec execute 358 sec latency 58.369 ms 2024-05-08T20:17:35.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 802910 9.01 MB/sec execute 359 sec latency 8.235 ms 2024-05-08T20:17:35.572 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1416507 16.11 MB/sec execute 359 sec latency 58.133 ms 2024-05-08T20:17:36.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 804555 9.00 MB/sec execute 360 sec latency 17.243 ms 2024-05-08T20:17:36.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:36 smithi188 ceph-mon[48761]: pgmap v545: 129 pgs: 129 active+clean; 195 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 3.9 MiB/s wr, 195 op/s 2024-05-08T20:17:36.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:36 smithi083 ceph-mon[38695]: pgmap v545: 129 pgs: 129 active+clean; 195 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 3.9 MiB/s wr, 195 op/s 2024-05-08T20:17:36.572 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1419267 16.11 MB/sec execute 360 sec latency 58.524 ms 2024-05-08T20:17:37.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 806249 9.01 MB/sec execute 361 sec latency 9.321 ms 2024-05-08T20:17:37.572 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1422063 16.11 MB/sec execute 361 sec latency 58.794 ms 2024-05-08T20:17:38.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 807677 9.00 MB/sec execute 362 sec latency 13.951 ms 2024-05-08T20:17:38.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:38 smithi188 ceph-mon[48761]: pgmap v546: 129 pgs: 129 active+clean; 195 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.1 MiB/s wr, 195 op/s 2024-05-08T20:17:38.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:38 smithi083 ceph-mon[38695]: pgmap v546: 129 pgs: 129 active+clean; 195 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.1 MiB/s wr, 195 op/s 2024-05-08T20:17:38.572 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1425526 16.12 MB/sec execute 362 sec latency 49.141 ms 2024-05-08T20:17:39.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 809467 9.01 MB/sec execute 363 sec latency 9.890 ms 2024-05-08T20:17:39.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:17:39.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:17:39.572 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1429008 16.12 MB/sec execute 363 sec latency 66.752 ms 2024-05-08T20:17:40.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 810787 8.99 MB/sec execute 364 sec latency 9.181 ms 2024-05-08T20:17:40.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:40 smithi188 ceph-mon[48761]: pgmap v547: 129 pgs: 129 active+clean; 193 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 4.1 MiB/s wr, 200 op/s 2024-05-08T20:17:40.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:40 smithi083 ceph-mon[38695]: pgmap v547: 129 pgs: 129 active+clean; 193 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 4.1 MiB/s wr, 200 op/s 2024-05-08T20:17:40.572 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1432494 16.13 MB/sec execute 364 sec latency 58.390 ms 2024-05-08T20:17:41.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 812707 9.00 MB/sec execute 365 sec latency 11.858 ms 2024-05-08T20:17:41.572 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1435105 16.13 MB/sec execute 365 sec latency 66.720 ms 2024-05-08T20:17:42.241 INFO:tasks.workunit.client.1.smithi188.stdout: 1 813990 8.99 MB/sec execute 366 sec latency 18.990 ms 2024-05-08T20:17:42.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:42 smithi188 ceph-mon[48761]: pgmap v548: 129 pgs: 129 active+clean; 191 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.4 MiB/s wr, 209 op/s 2024-05-08T20:17:42.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:42 smithi083 ceph-mon[38695]: pgmap v548: 129 pgs: 129 active+clean; 191 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.4 MiB/s wr, 209 op/s 2024-05-08T20:17:42.572 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1438280 16.13 MB/sec execute 366 sec latency 59.194 ms 2024-05-08T20:17:43.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 815762 9.00 MB/sec execute 367 sec latency 12.041 ms 2024-05-08T20:17:43.573 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1439891 16.11 MB/sec execute 367 sec latency 108.551 ms 2024-05-08T20:17:44.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 817294 9.00 MB/sec execute 368 sec latency 13.236 ms 2024-05-08T20:17:44.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:44 smithi188 ceph-mon[48761]: pgmap v549: 129 pgs: 129 active+clean; 191 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.5 MiB/s wr, 163 op/s 2024-05-08T20:17:44.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:44 smithi083 ceph-mon[38695]: pgmap v549: 129 pgs: 129 active+clean; 191 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.5 MiB/s wr, 163 op/s 2024-05-08T20:17:44.573 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1443262 16.11 MB/sec execute 368 sec latency 60.618 ms 2024-05-08T20:17:45.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 818983 8.99 MB/sec execute 369 sec latency 13.552 ms 2024-05-08T20:17:45.573 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1446736 16.12 MB/sec execute 369 sec latency 61.746 ms 2024-05-08T20:17:46.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 820601 9.00 MB/sec execute 370 sec latency 7.469 ms 2024-05-08T20:17:46.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:46 smithi188 ceph-mon[48761]: pgmap v550: 129 pgs: 129 active+clean; 193 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.3 MiB/s wr, 222 op/s 2024-05-08T20:17:46.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:46 smithi083 ceph-mon[38695]: pgmap v550: 129 pgs: 129 active+clean; 193 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.3 MiB/s wr, 222 op/s 2024-05-08T20:17:46.573 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1450261 16.13 MB/sec execute 370 sec latency 56.143 ms 2024-05-08T20:17:47.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 822217 8.99 MB/sec execute 371 sec latency 22.718 ms 2024-05-08T20:17:47.573 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1452902 16.12 MB/sec execute 371 sec latency 63.077 ms 2024-05-08T20:17:48.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 823969 9.00 MB/sec execute 372 sec latency 10.660 ms 2024-05-08T20:17:48.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:48 smithi188 ceph-mon[48761]: pgmap v551: 129 pgs: 129 active+clean; 192 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 184 op/s 2024-05-08T20:17:48.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:48 smithi083 ceph-mon[38695]: pgmap v551: 129 pgs: 129 active+clean; 192 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 184 op/s 2024-05-08T20:17:48.573 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1456097 16.13 MB/sec execute 372 sec latency 47.660 ms 2024-05-08T20:17:49.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 825487 8.99 MB/sec execute 373 sec latency 12.224 ms 2024-05-08T20:17:49.573 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1459045 16.12 MB/sec execute 373 sec latency 58.461 ms 2024-05-08T20:17:50.092 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:50 smithi188 ceph-mon[48761]: pgmap v552: 129 pgs: 129 active+clean; 194 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.9 MiB/s wr, 184 op/s 2024-05-08T20:17:50.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 827335 9.00 MB/sec execute 374 sec latency 5.289 ms 2024-05-08T20:17:50.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:50 smithi083 ceph-mon[38695]: pgmap v552: 129 pgs: 129 active+clean; 194 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.9 MiB/s wr, 184 op/s 2024-05-08T20:17:50.573 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1462100 16.12 MB/sec execute 374 sec latency 58.494 ms 2024-05-08T20:17:51.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 828786 8.99 MB/sec execute 375 sec latency 9.179 ms 2024-05-08T20:17:51.573 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1464751 16.12 MB/sec execute 375 sec latency 59.633 ms 2024-05-08T20:17:52.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 830568 8.99 MB/sec execute 376 sec latency 5.216 ms 2024-05-08T20:17:52.342 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:52 smithi188 ceph-mon[48761]: pgmap v553: 129 pgs: 129 active+clean; 193 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.2 MiB/s wr, 191 op/s 2024-05-08T20:17:52.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:52 smithi083 ceph-mon[38695]: pgmap v553: 129 pgs: 129 active+clean; 193 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.2 MiB/s wr, 191 op/s 2024-05-08T20:17:52.573 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1468188 16.13 MB/sec execute 376 sec latency 51.180 ms 2024-05-08T20:17:53.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 832060 8.99 MB/sec execute 377 sec latency 12.354 ms 2024-05-08T20:17:53.573 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1471593 16.13 MB/sec execute 377 sec latency 95.607 ms 2024-05-08T20:17:54.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 834050 9.00 MB/sec execute 378 sec latency 20.717 ms 2024-05-08T20:17:54.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:54 smithi188 ceph-mon[48761]: pgmap v554: 129 pgs: 129 active+clean; 193 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.4 MiB/s wr, 169 op/s 2024-05-08T20:17:54.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:17:54.344 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:54 smithi083 ceph-mon[38695]: pgmap v554: 129 pgs: 129 active+clean; 193 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.4 MiB/s wr, 169 op/s 2024-05-08T20:17:54.344 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:17:54.573 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1474185 16.13 MB/sec execute 378 sec latency 188.993 ms 2024-05-08T20:17:55.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 835373 8.99 MB/sec execute 379 sec latency 14.136 ms 2024-05-08T20:17:55.573 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1477389 16.13 MB/sec execute 379 sec latency 74.871 ms 2024-05-08T20:17:56.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 837126 8.99 MB/sec execute 380 sec latency 13.402 ms 2024-05-08T20:17:56.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:56 smithi188 ceph-mon[48761]: pgmap v555: 129 pgs: 129 active+clean; 195 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.1 MiB/s wr, 215 op/s 2024-05-08T20:17:56.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:56 smithi083 ceph-mon[38695]: pgmap v555: 129 pgs: 129 active+clean; 195 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.1 MiB/s wr, 215 op/s 2024-05-08T20:17:56.573 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1480861 16.14 MB/sec execute 380 sec latency 52.710 ms 2024-05-08T20:17:57.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 838585 8.99 MB/sec execute 381 sec latency 11.504 ms 2024-05-08T20:17:57.573 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1483961 16.13 MB/sec execute 381 sec latency 53.569 ms 2024-05-08T20:17:58.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 840246 8.98 MB/sec execute 382 sec latency 33.491 ms 2024-05-08T20:17:58.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:17:58 smithi188 ceph-mon[48761]: pgmap v556: 129 pgs: 129 active+clean; 196 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.2 MiB/s wr, 175 op/s 2024-05-08T20:17:58.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:17:58 smithi083 ceph-mon[38695]: pgmap v556: 129 pgs: 129 active+clean; 196 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.2 MiB/s wr, 175 op/s 2024-05-08T20:17:58.573 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1486926 16.14 MB/sec execute 382 sec latency 58.253 ms 2024-05-08T20:17:59.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 841827 8.99 MB/sec execute 383 sec latency 10.647 ms 2024-05-08T20:17:59.573 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1490466 16.14 MB/sec execute 383 sec latency 49.683 ms 2024-05-08T20:18:00.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 843294 8.98 MB/sec execute 384 sec latency 15.197 ms 2024-05-08T20:18:00.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:00 smithi188 ceph-mon[48761]: pgmap v557: 129 pgs: 129 active+clean; 200 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.3 MiB/s wr, 209 op/s 2024-05-08T20:18:00.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:00 smithi083 ceph-mon[38695]: pgmap v557: 129 pgs: 129 active+clean; 200 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.3 MiB/s wr, 209 op/s 2024-05-08T20:18:00.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1493610 16.15 MB/sec execute 384 sec latency 69.548 ms 2024-05-08T20:18:01.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 844850 8.98 MB/sec execute 385 sec latency 14.726 ms 2024-05-08T20:18:01.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:01 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:18:01.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:01 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:18:01.573 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1496403 16.15 MB/sec execute 385 sec latency 71.882 ms 2024-05-08T20:18:02.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 846128 8.98 MB/sec execute 386 sec latency 10.896 ms 2024-05-08T20:18:02.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:02 smithi188 ceph-mon[48761]: pgmap v558: 129 pgs: 129 active+clean; 194 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.4 MiB/s wr, 214 op/s 2024-05-08T20:18:02.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:02 smithi083 ceph-mon[38695]: pgmap v558: 129 pgs: 129 active+clean; 194 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.4 MiB/s wr, 214 op/s 2024-05-08T20:18:02.573 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1498584 16.14 MB/sec execute 386 sec latency 65.998 ms 2024-05-08T20:18:03.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 847621 8.97 MB/sec execute 387 sec latency 15.309 ms 2024-05-08T20:18:03.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1500101 16.11 MB/sec execute 387 sec latency 72.354 ms 2024-05-08T20:18:04.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:03 smithi083 ceph-mon[38695]: pgmap v559: 129 pgs: 129 active+clean; 194 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.3 MiB/s wr, 154 op/s 2024-05-08T20:18:04.193 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:03 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:18:04.193 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:03 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:18:04.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 849140 8.98 MB/sec execute 388 sec latency 7.262 ms 2024-05-08T20:18:04.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:03 smithi188 ceph-mon[48761]: pgmap v559: 129 pgs: 129 active+clean; 194 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.3 MiB/s wr, 154 op/s 2024-05-08T20:18:04.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:03 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:18:04.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:03 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:18:04.573 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1501697 16.09 MB/sec execute 388 sec latency 348.804 ms 2024-05-08T20:18:05.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 850721 8.97 MB/sec execute 389 sec latency 23.311 ms 2024-05-08T20:18:05.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1502511 16.06 MB/sec execute 389 sec latency 678.671 ms 2024-05-08T20:18:06.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:05 smithi083 ceph-mon[38695]: pgmap v560: 129 pgs: 129 active+clean; 196 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.0 MiB/s wr, 207 op/s 2024-05-08T20:18:06.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 852385 8.98 MB/sec execute 390 sec latency 12.036 ms 2024-05-08T20:18:06.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:05 smithi188 ceph-mon[48761]: pgmap v560: 129 pgs: 129 active+clean; 196 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.0 MiB/s wr, 207 op/s 2024-05-08T20:18:06.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1503749 16.04 MB/sec execute 390 sec latency 62.989 ms 2024-05-08T20:18:07.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 853758 8.97 MB/sec execute 391 sec latency 17.669 ms 2024-05-08T20:18:07.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1506990 16.04 MB/sec execute 391 sec latency 60.481 ms 2024-05-08T20:18:08.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:07 smithi083 ceph-mon[38695]: pgmap v561: 129 pgs: 129 active+clean; 191 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.0 MiB/s wr, 173 op/s 2024-05-08T20:18:08.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 855439 8.97 MB/sec execute 392 sec latency 6.382 ms 2024-05-08T20:18:08.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:07 smithi188 ceph-mon[48761]: pgmap v561: 129 pgs: 129 active+clean; 191 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.0 MiB/s wr, 173 op/s 2024-05-08T20:18:08.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1509634 16.04 MB/sec execute 392 sec latency 48.906 ms 2024-05-08T20:18:09.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:18:09.193 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:18:09.193 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:18:09.193 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:18:09.193 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:18:09.193 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:18:09.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 856761 8.96 MB/sec execute 393 sec latency 14.750 ms 2024-05-08T20:18:09.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:18:09.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:18:09.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:18:09.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:18:09.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:18:09.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:18:09.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1512831 16.04 MB/sec execute 393 sec latency 58.403 ms 2024-05-08T20:18:10.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:09 smithi083 ceph-mon[38695]: pgmap v562: 129 pgs: 129 active+clean; 195 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 3.8 MiB/s wr, 202 op/s 2024-05-08T20:18:10.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 858526 8.96 MB/sec execute 394 sec latency 15.185 ms 2024-05-08T20:18:10.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:09 smithi188 ceph-mon[48761]: pgmap v562: 129 pgs: 129 active+clean; 195 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 3.8 MiB/s wr, 202 op/s 2024-05-08T20:18:10.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1515817 16.04 MB/sec execute 394 sec latency 56.532 ms 2024-05-08T20:18:11.242 INFO:tasks.workunit.client.1.smithi188.stdout: 1 859942 8.96 MB/sec execute 395 sec latency 9.759 ms 2024-05-08T20:18:11.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1518835 16.04 MB/sec execute 395 sec latency 48.488 ms 2024-05-08T20:18:12.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:11 smithi083 ceph-mon[38695]: pgmap v563: 129 pgs: 129 active+clean; 195 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.7 MiB/s wr, 173 op/s 2024-05-08T20:18:12.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 861626 8.96 MB/sec execute 396 sec latency 14.114 ms 2024-05-08T20:18:12.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:11 smithi188 ceph-mon[48761]: pgmap v563: 129 pgs: 129 active+clean; 195 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.7 MiB/s wr, 173 op/s 2024-05-08T20:18:12.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1521481 16.04 MB/sec execute 396 sec latency 57.448 ms 2024-05-08T20:18:13.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 863235 8.96 MB/sec execute 397 sec latency 8.275 ms 2024-05-08T20:18:13.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1524845 16.04 MB/sec execute 397 sec latency 58.197 ms 2024-05-08T20:18:14.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:13 smithi083 ceph-mon[38695]: pgmap v564: 129 pgs: 129 active+clean; 195 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.1 MiB/s wr, 153 op/s 2024-05-08T20:18:14.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 864885 8.96 MB/sec execute 398 sec latency 22.727 ms 2024-05-08T20:18:14.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:13 smithi188 ceph-mon[48761]: pgmap v564: 129 pgs: 129 active+clean; 195 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.1 MiB/s wr, 153 op/s 2024-05-08T20:18:14.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1528327 16.05 MB/sec execute 398 sec latency 51.971 ms 2024-05-08T20:18:15.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 866597 8.97 MB/sec execute 399 sec latency 8.069 ms 2024-05-08T20:18:15.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1531813 16.06 MB/sec execute 399 sec latency 79.506 ms 2024-05-08T20:18:16.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:15 smithi083 ceph-mon[38695]: pgmap v565: 129 pgs: 129 active+clean; 192 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 3.8 MiB/s wr, 207 op/s 2024-05-08T20:18:16.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 868098 8.96 MB/sec execute 400 sec latency 11.902 ms 2024-05-08T20:18:16.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:15 smithi188 ceph-mon[48761]: pgmap v565: 129 pgs: 129 active+clean; 192 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 3.8 MiB/s wr, 207 op/s 2024-05-08T20:18:16.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1534427 16.05 MB/sec execute 400 sec latency 154.441 ms 2024-05-08T20:18:17.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 869860 8.97 MB/sec execute 401 sec latency 13.375 ms 2024-05-08T20:18:17.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1537593 16.06 MB/sec execute 401 sec latency 53.164 ms 2024-05-08T20:18:18.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:17 smithi083 ceph-mon[38695]: pgmap v566: 129 pgs: 129 active+clean; 194 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.9 MiB/s wr, 170 op/s 2024-05-08T20:18:18.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 871323 8.96 MB/sec execute 402 sec latency 16.909 ms 2024-05-08T20:18:18.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:17 smithi188 ceph-mon[48761]: pgmap v566: 129 pgs: 129 active+clean; 194 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.9 MiB/s wr, 170 op/s 2024-05-08T20:18:18.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1540592 16.05 MB/sec execute 402 sec latency 61.483 ms 2024-05-08T20:18:19.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 873151 8.96 MB/sec execute 403 sec latency 15.121 ms 2024-05-08T20:18:19.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1543252 16.05 MB/sec execute 403 sec latency 60.870 ms 2024-05-08T20:18:19.942 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:19 smithi083 ceph-mon[38695]: pgmap v567: 129 pgs: 129 active+clean; 198 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 4.0 MiB/s wr, 210 op/s 2024-05-08T20:18:20.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 874541 8.96 MB/sec execute 404 sec latency 9.708 ms 2024-05-08T20:18:20.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:19 smithi188 ceph-mon[48761]: pgmap v567: 129 pgs: 129 active+clean; 198 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 4.0 MiB/s wr, 210 op/s 2024-05-08T20:18:20.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1546191 16.05 MB/sec execute 404 sec latency 58.446 ms 2024-05-08T20:18:21.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 876376 8.96 MB/sec execute 405 sec latency 9.820 ms 2024-05-08T20:18:21.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1549657 16.06 MB/sec execute 405 sec latency 58.428 ms 2024-05-08T20:18:22.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:21 smithi083 ceph-mon[38695]: pgmap v568: 129 pgs: 129 active+clean; 195 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.2 MiB/s wr, 183 op/s 2024-05-08T20:18:22.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 877812 8.96 MB/sec execute 406 sec latency 15.539 ms 2024-05-08T20:18:22.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:21 smithi188 ceph-mon[48761]: pgmap v568: 129 pgs: 129 active+clean; 195 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.2 MiB/s wr, 183 op/s 2024-05-08T20:18:22.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1553132 16.06 MB/sec execute 406 sec latency 58.298 ms 2024-05-08T20:18:23.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 879492 8.95 MB/sec execute 407 sec latency 20.810 ms 2024-05-08T20:18:23.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1556623 16.07 MB/sec execute 407 sec latency 58.332 ms 2024-05-08T20:18:24.193 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:23 smithi083 ceph-mon[38695]: pgmap v569: 129 pgs: 129 active+clean; 196 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.5 MiB/s wr, 161 op/s 2024-05-08T20:18:24.193 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:18:24.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 881075 8.96 MB/sec execute 408 sec latency 14.955 ms 2024-05-08T20:18:24.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:23 smithi188 ceph-mon[48761]: pgmap v569: 129 pgs: 129 active+clean; 196 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.5 MiB/s wr, 161 op/s 2024-05-08T20:18:24.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:18:24.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1559247 16.07 MB/sec execute 408 sec latency 58.473 ms 2024-05-08T20:18:25.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 882784 8.95 MB/sec execute 409 sec latency 22.906 ms 2024-05-08T20:18:25.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1561966 16.06 MB/sec execute 409 sec latency 58.326 ms 2024-05-08T20:18:26.193 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:25 smithi083 ceph-mon[38695]: pgmap v570: 129 pgs: 129 active+clean; 198 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 4.2 MiB/s wr, 213 op/s 2024-05-08T20:18:26.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 884434 8.96 MB/sec execute 410 sec latency 6.836 ms 2024-05-08T20:18:26.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:25 smithi188 ceph-mon[48761]: pgmap v570: 129 pgs: 129 active+clean; 198 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 4.2 MiB/s wr, 213 op/s 2024-05-08T20:18:26.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1564676 16.06 MB/sec execute 410 sec latency 91.622 ms 2024-05-08T20:18:27.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 886072 8.95 MB/sec execute 411 sec latency 10.166 ms 2024-05-08T20:18:27.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1567463 16.06 MB/sec execute 411 sec latency 60.728 ms 2024-05-08T20:18:28.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:27 smithi083 ceph-mon[38695]: pgmap v571: 129 pgs: 129 active+clean; 192 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.2 MiB/s wr, 180 op/s 2024-05-08T20:18:28.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 887847 8.96 MB/sec execute 412 sec latency 15.149 ms 2024-05-08T20:18:28.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:27 smithi188 ceph-mon[48761]: pgmap v571: 129 pgs: 129 active+clean; 192 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.2 MiB/s wr, 180 op/s 2024-05-08T20:18:28.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1570873 16.06 MB/sec execute 412 sec latency 60.941 ms 2024-05-08T20:18:29.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 889367 8.95 MB/sec execute 413 sec latency 7.840 ms 2024-05-08T20:18:29.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1574356 16.07 MB/sec execute 413 sec latency 59.575 ms 2024-05-08T20:18:30.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:29 smithi083 ceph-mon[38695]: pgmap v572: 129 pgs: 129 active+clean; 193 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.0 MiB/s wr, 189 op/s 2024-05-08T20:18:30.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 891183 8.96 MB/sec execute 414 sec latency 8.856 ms 2024-05-08T20:18:30.262 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:29 smithi188 ceph-mon[48761]: pgmap v572: 129 pgs: 129 active+clean; 193 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.0 MiB/s wr, 189 op/s 2024-05-08T20:18:30.574 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1577359 16.07 MB/sec execute 414 sec latency 60.769 ms 2024-05-08T20:18:31.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 892606 8.96 MB/sec execute 415 sec latency 9.587 ms 2024-05-08T20:18:31.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1580139 16.07 MB/sec execute 415 sec latency 57.710 ms 2024-05-08T20:18:32.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:31 smithi083 ceph-mon[38695]: pgmap v573: 129 pgs: 129 active+clean; 196 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.3 MiB/s wr, 198 op/s 2024-05-08T20:18:32.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 894418 8.96 MB/sec execute 416 sec latency 8.634 ms 2024-05-08T20:18:32.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:31 smithi188 ceph-mon[48761]: pgmap v573: 129 pgs: 129 active+clean; 196 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.3 MiB/s wr, 198 op/s 2024-05-08T20:18:32.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1582655 16.06 MB/sec execute 416 sec latency 55.615 ms 2024-05-08T20:18:33.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 895766 8.95 MB/sec execute 417 sec latency 9.724 ms 2024-05-08T20:18:33.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1585950 16.06 MB/sec execute 417 sec latency 55.711 ms 2024-05-08T20:18:34.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:33 smithi083 ceph-mon[38695]: pgmap v574: 129 pgs: 129 active+clean; 192 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.4 MiB/s wr, 177 op/s 2024-05-08T20:18:34.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 897572 8.96 MB/sec execute 418 sec latency 10.319 ms 2024-05-08T20:18:34.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:33 smithi188 ceph-mon[48761]: pgmap v574: 129 pgs: 129 active+clean; 192 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.4 MiB/s wr, 177 op/s 2024-05-08T20:18:34.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1588777 16.06 MB/sec execute 418 sec latency 58.436 ms 2024-05-08T20:18:35.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 898989 8.95 MB/sec execute 419 sec latency 16.630 ms 2024-05-08T20:18:35.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1592267 16.07 MB/sec execute 419 sec latency 58.611 ms 2024-05-08T20:18:36.193 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:35 smithi083 ceph-mon[38695]: pgmap v575: 129 pgs: 129 active+clean; 191 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.8 MiB/s wr, 193 op/s 2024-05-08T20:18:36.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 900607 8.95 MB/sec execute 420 sec latency 8.765 ms 2024-05-08T20:18:36.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:35 smithi188 ceph-mon[48761]: pgmap v575: 129 pgs: 129 active+clean; 191 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.8 MiB/s wr, 193 op/s 2024-05-08T20:18:36.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1593933 16.05 MB/sec execute 420 sec latency 358.499 ms 2024-05-08T20:18:37.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 902187 8.95 MB/sec execute 421 sec latency 24.280 ms 2024-05-08T20:18:37.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1595679 16.04 MB/sec execute 421 sec latency 593.215 ms 2024-05-08T20:18:38.193 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:37 smithi083 ceph-mon[38695]: pgmap v576: 129 pgs: 129 active+clean; 194 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.1 MiB/s wr, 192 op/s 2024-05-08T20:18:38.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 903737 8.94 MB/sec execute 422 sec latency 15.752 ms 2024-05-08T20:18:38.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:37 smithi188 ceph-mon[48761]: pgmap v576: 129 pgs: 129 active+clean; 194 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.1 MiB/s wr, 192 op/s 2024-05-08T20:18:38.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1598299 16.03 MB/sec execute 422 sec latency 56.708 ms 2024-05-08T20:18:39.193 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:18:39.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 905502 8.96 MB/sec execute 423 sec latency 12.898 ms 2024-05-08T20:18:39.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:18:39.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1601312 16.03 MB/sec execute 423 sec latency 50.106 ms 2024-05-08T20:18:40.193 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:39 smithi083 ceph-mon[38695]: pgmap v577: 129 pgs: 129 active+clean; 195 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 4.0 MiB/s wr, 185 op/s 2024-05-08T20:18:40.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 906836 8.95 MB/sec execute 424 sec latency 22.493 ms 2024-05-08T20:18:40.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:39 smithi188 ceph-mon[48761]: pgmap v577: 129 pgs: 129 active+clean; 195 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 4.0 MiB/s wr, 185 op/s 2024-05-08T20:18:40.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1603978 16.03 MB/sec execute 424 sec latency 60.503 ms 2024-05-08T20:18:41.243 INFO:tasks.workunit.client.1.smithi188.stdout: 1 908613 8.95 MB/sec execute 425 sec latency 14.857 ms 2024-05-08T20:18:41.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1606560 16.02 MB/sec execute 425 sec latency 60.509 ms 2024-05-08T20:18:42.193 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:41 smithi083 ceph-mon[38695]: pgmap v578: 129 pgs: 129 active+clean; 193 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.3 MiB/s wr, 220 op/s 2024-05-08T20:18:42.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 909985 8.94 MB/sec execute 426 sec latency 9.584 ms 2024-05-08T20:18:42.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:41 smithi188 ceph-mon[48761]: pgmap v578: 129 pgs: 129 active+clean; 193 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.3 MiB/s wr, 220 op/s 2024-05-08T20:18:42.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1609982 16.03 MB/sec execute 426 sec latency 61.260 ms 2024-05-08T20:18:43.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 911803 8.95 MB/sec execute 427 sec latency 8.604 ms 2024-05-08T20:18:43.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1613461 16.04 MB/sec execute 427 sec latency 61.447 ms 2024-05-08T20:18:44.192 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:43 smithi083 ceph-mon[38695]: pgmap v579: 129 pgs: 129 active+clean; 193 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.3 MiB/s wr, 157 op/s 2024-05-08T20:18:44.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 913194 8.94 MB/sec execute 428 sec latency 11.445 ms 2024-05-08T20:18:44.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:43 smithi188 ceph-mon[48761]: pgmap v579: 129 pgs: 129 active+clean; 193 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.3 MiB/s wr, 157 op/s 2024-05-08T20:18:44.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1616952 16.04 MB/sec execute 428 sec latency 49.417 ms 2024-05-08T20:18:45.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 914816 8.94 MB/sec execute 429 sec latency 22.285 ms 2024-05-08T20:18:45.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1619573 16.04 MB/sec execute 429 sec latency 61.688 ms 2024-05-08T20:18:46.193 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:45 smithi083 ceph-mon[38695]: pgmap v580: 129 pgs: 129 active+clean; 193 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.7 MiB/s wr, 169 op/s 2024-05-08T20:18:46.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 916397 8.94 MB/sec execute 430 sec latency 8.670 ms 2024-05-08T20:18:46.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:45 smithi188 ceph-mon[48761]: pgmap v580: 129 pgs: 129 active+clean; 193 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.7 MiB/s wr, 169 op/s 2024-05-08T20:18:46.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1622739 16.04 MB/sec execute 430 sec latency 53.847 ms 2024-05-08T20:18:47.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 917939 8.98 MB/sec execute 431 sec latency 26.141 ms 2024-05-08T20:18:47.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1626280 16.05 MB/sec execute 431 sec latency 53.066 ms 2024-05-08T20:18:48.193 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:47 smithi083 ceph-mon[38695]: pgmap v581: 129 pgs: 129 active+clean; 191 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.2 MiB/s wr, 214 op/s 2024-05-08T20:18:48.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 919892 8.98 MB/sec execute 432 sec latency 15.154 ms 2024-05-08T20:18:48.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:47 smithi188 ceph-mon[48761]: pgmap v581: 129 pgs: 129 active+clean; 191 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.2 MiB/s wr, 214 op/s 2024-05-08T20:18:48.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1629289 16.04 MB/sec execute 432 sec latency 107.681 ms 2024-05-08T20:18:49.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 921128 8.98 MB/sec execute 433 sec latency 26.797 ms 2024-05-08T20:18:49.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1632347 16.05 MB/sec execute 433 sec latency 59.160 ms 2024-05-08T20:18:50.193 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:49 smithi083 ceph-mon[38695]: pgmap v582: 129 pgs: 129 active+clean; 196 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.1 MiB/s wr, 173 op/s 2024-05-08T20:18:50.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 922752 8.97 MB/sec execute 434 sec latency 13.472 ms 2024-05-08T20:18:50.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:49 smithi188 ceph-mon[48761]: pgmap v582: 129 pgs: 129 active+clean; 196 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.1 MiB/s wr, 173 op/s 2024-05-08T20:18:50.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1635895 16.05 MB/sec execute 434 sec latency 58.409 ms 2024-05-08T20:18:51.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 924323 8.98 MB/sec execute 435 sec latency 8.105 ms 2024-05-08T20:18:51.575 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1638537 16.05 MB/sec execute 435 sec latency 58.455 ms 2024-05-08T20:18:52.193 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:51 smithi083 ceph-mon[38695]: pgmap v583: 129 pgs: 129 active+clean; 201 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 5.9 MiB/s wr, 220 op/s 2024-05-08T20:18:52.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 925960 8.97 MB/sec execute 436 sec latency 17.063 ms 2024-05-08T20:18:52.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:51 smithi188 ceph-mon[48761]: pgmap v583: 129 pgs: 129 active+clean; 201 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 5.9 MiB/s wr, 220 op/s 2024-05-08T20:18:52.576 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1641905 16.06 MB/sec execute 436 sec latency 61.786 ms 2024-05-08T20:18:53.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 927656 8.98 MB/sec execute 437 sec latency 13.451 ms 2024-05-08T20:18:53.576 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1645376 16.06 MB/sec execute 437 sec latency 58.370 ms 2024-05-08T20:18:54.193 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:53 smithi083 ceph-mon[38695]: pgmap v584: 129 pgs: 129 active+clean; 196 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 5.0 MiB/s wr, 165 op/s 2024-05-08T20:18:54.193 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:53 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:18:54.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 929149 8.97 MB/sec execute 438 sec latency 21.857 ms 2024-05-08T20:18:54.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:53 smithi188 ceph-mon[48761]: pgmap v584: 129 pgs: 129 active+clean; 196 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 5.0 MiB/s wr, 165 op/s 2024-05-08T20:18:54.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:53 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:18:54.576 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1648865 16.07 MB/sec execute 438 sec latency 75.129 ms 2024-05-08T20:18:55.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 930947 8.98 MB/sec execute 439 sec latency 8.454 ms 2024-05-08T20:18:55.576 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1651485 16.06 MB/sec execute 439 sec latency 46.560 ms 2024-05-08T20:18:56.193 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:55 smithi083 ceph-mon[38695]: pgmap v585: 129 pgs: 129 active+clean; 193 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.5 MiB/s wr, 212 op/s 2024-05-08T20:18:56.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 932338 8.97 MB/sec execute 440 sec latency 15.309 ms 2024-05-08T20:18:56.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:55 smithi188 ceph-mon[48761]: pgmap v585: 129 pgs: 129 active+clean; 193 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.5 MiB/s wr, 212 op/s 2024-05-08T20:18:56.576 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1654492 16.06 MB/sec execute 440 sec latency 58.411 ms 2024-05-08T20:18:57.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 934179 8.97 MB/sec execute 441 sec latency 11.082 ms 2024-05-08T20:18:57.576 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1657166 16.06 MB/sec execute 441 sec latency 58.190 ms 2024-05-08T20:18:58.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 935486 8.97 MB/sec execute 442 sec latency 8.379 ms 2024-05-08T20:18:58.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:57 smithi188 ceph-mon[48761]: pgmap v586: 129 pgs: 129 active+clean; 195 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 6.1 MiB/s wr, 213 op/s 2024-05-08T20:18:58.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:57 smithi083 ceph-mon[38695]: pgmap v586: 129 pgs: 129 active+clean; 195 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 6.1 MiB/s wr, 213 op/s 2024-05-08T20:18:58.576 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1659629 16.05 MB/sec execute 442 sec latency 87.149 ms 2024-05-08T20:18:59.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 937228 8.97 MB/sec execute 443 sec latency 27.782 ms 2024-05-08T20:18:59.576 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1663109 16.06 MB/sec execute 443 sec latency 57.256 ms 2024-05-08T20:19:00.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 938734 8.97 MB/sec execute 444 sec latency 10.891 ms 2024-05-08T20:19:00.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:18:59 smithi188 ceph-mon[48761]: pgmap v587: 129 pgs: 129 active+clean; 197 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 5.5 MiB/s wr, 205 op/s 2024-05-08T20:19:00.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:18:59 smithi083 ceph-mon[38695]: pgmap v587: 129 pgs: 129 active+clean; 197 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 5.5 MiB/s wr, 205 op/s 2024-05-08T20:19:00.576 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1666596 16.07 MB/sec execute 444 sec latency 58.413 ms 2024-05-08T20:19:01.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 940382 8.96 MB/sec execute 445 sec latency 17.145 ms 2024-05-08T20:19:01.576 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1669243 16.06 MB/sec execute 445 sec latency 58.273 ms 2024-05-08T20:19:02.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 941994 8.97 MB/sec execute 446 sec latency 7.674 ms 2024-05-08T20:19:02.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:01 smithi188 ceph-mon[48761]: pgmap v588: 129 pgs: 129 active+clean; 198 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 5.7 MiB/s wr, 219 op/s 2024-05-08T20:19:02.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:01 smithi083 ceph-mon[38695]: pgmap v588: 129 pgs: 129 active+clean; 198 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 5.7 MiB/s wr, 219 op/s 2024-05-08T20:19:02.576 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1672443 16.07 MB/sec execute 446 sec latency 58.160 ms 2024-05-08T20:19:03.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 943564 8.96 MB/sec execute 447 sec latency 23.538 ms 2024-05-08T20:19:03.576 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1675353 16.06 MB/sec execute 447 sec latency 58.197 ms 2024-05-08T20:19:04.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 945332 8.97 MB/sec execute 448 sec latency 10.385 ms 2024-05-08T20:19:04.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:03 smithi188 ceph-mon[48761]: pgmap v589: 129 pgs: 129 active+clean; 197 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.4 MiB/s wr, 161 op/s 2024-05-08T20:19:04.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:03 smithi083 ceph-mon[38695]: pgmap v589: 129 pgs: 129 active+clean; 197 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.4 MiB/s wr, 161 op/s 2024-05-08T20:19:04.576 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1677547 16.05 MB/sec execute 448 sec latency 55.856 ms 2024-05-08T20:19:05.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 946706 8.96 MB/sec execute 449 sec latency 16.966 ms 2024-05-08T20:19:05.576 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1680910 16.06 MB/sec execute 449 sec latency 58.523 ms 2024-05-08T20:19:06.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 948438 8.97 MB/sec execute 450 sec latency 6.187 ms 2024-05-08T20:19:06.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:05 smithi188 ceph-mon[48761]: pgmap v590: 129 pgs: 129 active+clean; 197 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 3.9 MiB/s wr, 181 op/s 2024-05-08T20:19:06.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:05 smithi083 ceph-mon[38695]: pgmap v590: 129 pgs: 129 active+clean; 197 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 3.9 MiB/s wr, 181 op/s 2024-05-08T20:19:06.576 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1684386 16.07 MB/sec execute 450 sec latency 50.764 ms 2024-05-08T20:19:07.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 949873 8.96 MB/sec execute 451 sec latency 8.058 ms 2024-05-08T20:19:07.576 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1687876 16.07 MB/sec execute 451 sec latency 59.695 ms 2024-05-08T20:19:08.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 951817 8.96 MB/sec execute 452 sec latency 11.643 ms 2024-05-08T20:19:08.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:07 smithi188 ceph-mon[48761]: pgmap v591: 129 pgs: 129 active+clean; 197 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.3 MiB/s wr, 188 op/s 2024-05-08T20:19:08.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:07 smithi083 ceph-mon[38695]: pgmap v591: 129 pgs: 129 active+clean; 197 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.3 MiB/s wr, 188 op/s 2024-05-08T20:19:08.576 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1689534 16.05 MB/sec execute 452 sec latency 247.032 ms 2024-05-08T20:19:09.244 INFO:tasks.workunit.client.1.smithi188.stdout: 1 952986 8.96 MB/sec execute 453 sec latency 19.073 ms 2024-05-08T20:19:09.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:19:09.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:19:09.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:19:09.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:19:09.576 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1689605 16.02 MB/sec execute 453 sec latency 546.888 ms 2024-05-08T20:19:10.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 954607 8.95 MB/sec execute 454 sec latency 17.231 ms 2024-05-08T20:19:10.319 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:09 smithi083 ceph-mon[38695]: pgmap v592: 129 pgs: 129 active+clean; 195 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.5 MiB/s wr, 186 op/s 2024-05-08T20:19:10.319 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:19:10.319 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:19:10.319 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:19:10.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:09 smithi188 ceph-mon[48761]: pgmap v592: 129 pgs: 129 active+clean; 195 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.5 MiB/s wr, 186 op/s 2024-05-08T20:19:10.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:19:10.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:19:10.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:19:10.576 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1691542 16.01 MB/sec execute 454 sec latency 574.913 ms 2024-05-08T20:19:11.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 956205 8.96 MB/sec execute 455 sec latency 14.694 ms 2024-05-08T20:19:11.576 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1695013 16.01 MB/sec execute 455 sec latency 58.095 ms 2024-05-08T20:19:12.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 957809 8.95 MB/sec execute 456 sec latency 30.873 ms 2024-05-08T20:19:12.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:11 smithi188 ceph-mon[48761]: pgmap v593: 129 pgs: 129 active+clean; 195 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.4 MiB/s wr, 194 op/s 2024-05-08T20:19:12.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:11 smithi083 ceph-mon[38695]: pgmap v593: 129 pgs: 129 active+clean; 195 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.4 MiB/s wr, 194 op/s 2024-05-08T20:19:12.576 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1698505 16.02 MB/sec execute 456 sec latency 66.688 ms 2024-05-08T20:19:13.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 959550 8.96 MB/sec execute 457 sec latency 9.489 ms 2024-05-08T20:19:13.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1701128 16.02 MB/sec execute 457 sec latency 49.711 ms 2024-05-08T20:19:14.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 960923 8.96 MB/sec execute 458 sec latency 18.061 ms 2024-05-08T20:19:14.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:13 smithi188 ceph-mon[48761]: pgmap v594: 129 pgs: 129 active+clean; 195 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.3 MiB/s wr, 169 op/s 2024-05-08T20:19:14.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:13 smithi083 ceph-mon[38695]: pgmap v594: 129 pgs: 129 active+clean; 195 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.3 MiB/s wr, 169 op/s 2024-05-08T20:19:14.576 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1704282 16.02 MB/sec execute 458 sec latency 66.727 ms 2024-05-08T20:19:15.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 962643 8.96 MB/sec execute 459 sec latency 10.679 ms 2024-05-08T20:19:15.576 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1706811 16.01 MB/sec execute 459 sec latency 58.329 ms 2024-05-08T20:19:16.093 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:15 smithi188 ceph-mon[48761]: pgmap v595: 129 pgs: 129 active+clean; 195 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.7 MiB/s wr, 180 op/s 2024-05-08T20:19:16.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 964032 8.95 MB/sec execute 460 sec latency 13.447 ms 2024-05-08T20:19:16.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:15 smithi083 ceph-mon[38695]: pgmap v595: 129 pgs: 129 active+clean; 195 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.7 MiB/s wr, 180 op/s 2024-05-08T20:19:16.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1709453 16.01 MB/sec execute 460 sec latency 52.534 ms 2024-05-08T20:19:17.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 965831 8.95 MB/sec execute 461 sec latency 10.971 ms 2024-05-08T20:19:17.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1712821 16.01 MB/sec execute 461 sec latency 58.305 ms 2024-05-08T20:19:18.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 967203 8.95 MB/sec execute 462 sec latency 19.180 ms 2024-05-08T20:19:18.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:17 smithi188 ceph-mon[48761]: pgmap v596: 129 pgs: 129 active+clean; 194 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.1 MiB/s wr, 213 op/s 2024-05-08T20:19:18.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:17 smithi083 ceph-mon[38695]: pgmap v596: 129 pgs: 129 active+clean; 194 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.1 MiB/s wr, 213 op/s 2024-05-08T20:19:18.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1716295 16.02 MB/sec execute 462 sec latency 58.207 ms 2024-05-08T20:19:19.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 968820 8.95 MB/sec execute 463 sec latency 12.457 ms 2024-05-08T20:19:19.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1719832 16.03 MB/sec execute 463 sec latency 49.701 ms 2024-05-08T20:19:20.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 970382 8.95 MB/sec execute 464 sec latency 9.111 ms 2024-05-08T20:19:20.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:19 smithi188 ceph-mon[48761]: pgmap v597: 129 pgs: 129 active+clean; 193 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 3.3 MiB/s wr, 168 op/s 2024-05-08T20:19:20.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:19 smithi083 ceph-mon[38695]: pgmap v597: 129 pgs: 129 active+clean; 193 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 3.3 MiB/s wr, 168 op/s 2024-05-08T20:19:20.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1722392 16.02 MB/sec execute 464 sec latency 88.805 ms 2024-05-08T20:19:21.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 971959 8.94 MB/sec execute 465 sec latency 11.125 ms 2024-05-08T20:19:21.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1725639 16.03 MB/sec execute 465 sec latency 50.681 ms 2024-05-08T20:19:22.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 973717 8.95 MB/sec execute 466 sec latency 13.219 ms 2024-05-08T20:19:22.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:21 smithi188 ceph-mon[48761]: pgmap v598: 129 pgs: 129 active+clean; 198 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.5 MiB/s wr, 215 op/s 2024-05-08T20:19:22.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:21 smithi083 ceph-mon[38695]: pgmap v598: 129 pgs: 129 active+clean; 198 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.5 MiB/s wr, 215 op/s 2024-05-08T20:19:22.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1728537 16.02 MB/sec execute 466 sec latency 91.711 ms 2024-05-08T20:19:23.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 975136 8.95 MB/sec execute 467 sec latency 14.147 ms 2024-05-08T20:19:23.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1731220 16.02 MB/sec execute 467 sec latency 51.506 ms 2024-05-08T20:19:24.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 976978 8.95 MB/sec execute 468 sec latency 6.501 ms 2024-05-08T20:19:24.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:23 smithi188 ceph-mon[48761]: pgmap v599: 129 pgs: 129 active+clean; 197 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.3 MiB/s wr, 158 op/s 2024-05-08T20:19:24.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:19:24.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:23 smithi083 ceph-mon[38695]: pgmap v599: 129 pgs: 129 active+clean; 197 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.3 MiB/s wr, 158 op/s 2024-05-08T20:19:24.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:19:24.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1734165 16.02 MB/sec execute 468 sec latency 58.766 ms 2024-05-08T20:19:25.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 978418 8.95 MB/sec execute 469 sec latency 15.290 ms 2024-05-08T20:19:25.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1737631 16.03 MB/sec execute 469 sec latency 57.602 ms 2024-05-08T20:19:26.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 980303 8.95 MB/sec execute 470 sec latency 11.225 ms 2024-05-08T20:19:26.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:25 smithi188 ceph-mon[48761]: pgmap v600: 129 pgs: 129 active+clean; 197 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.6 MiB/s wr, 170 op/s 2024-05-08T20:19:26.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:25 smithi083 ceph-mon[38695]: pgmap v600: 129 pgs: 129 active+clean; 197 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.6 MiB/s wr, 170 op/s 2024-05-08T20:19:26.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1741113 16.03 MB/sec execute 470 sec latency 62.189 ms 2024-05-08T20:19:27.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 981598 8.94 MB/sec execute 471 sec latency 14.092 ms 2024-05-08T20:19:27.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1744650 16.04 MB/sec execute 471 sec latency 64.561 ms 2024-05-08T20:19:28.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 983316 8.95 MB/sec execute 472 sec latency 29.510 ms 2024-05-08T20:19:28.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:27 smithi188 ceph-mon[48761]: pgmap v601: 129 pgs: 129 active+clean; 199 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.3 MiB/s wr, 209 op/s 2024-05-08T20:19:28.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:27 smithi083 ceph-mon[38695]: pgmap v601: 129 pgs: 129 active+clean; 199 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.3 MiB/s wr, 209 op/s 2024-05-08T20:19:28.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1748149 16.05 MB/sec execute 472 sec latency 54.570 ms 2024-05-08T20:19:29.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 984797 8.94 MB/sec execute 473 sec latency 9.337 ms 2024-05-08T20:19:29.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1751688 16.05 MB/sec execute 473 sec latency 58.391 ms 2024-05-08T20:19:30.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 986446 8.94 MB/sec execute 474 sec latency 15.572 ms 2024-05-08T20:19:30.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:29 smithi188 ceph-mon[48761]: pgmap v602: 129 pgs: 129 active+clean; 197 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.5 MiB/s wr, 163 op/s 2024-05-08T20:19:30.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:29 smithi083 ceph-mon[38695]: pgmap v602: 129 pgs: 129 active+clean; 197 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.5 MiB/s wr, 163 op/s 2024-05-08T20:19:30.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1753812 16.04 MB/sec execute 474 sec latency 86.514 ms 2024-05-08T20:19:31.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 988034 8.95 MB/sec execute 475 sec latency 14.680 ms 2024-05-08T20:19:31.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1756808 16.04 MB/sec execute 475 sec latency 62.428 ms 2024-05-08T20:19:32.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 989513 8.94 MB/sec execute 476 sec latency 7.138 ms 2024-05-08T20:19:32.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:31 smithi188 ceph-mon[48761]: pgmap v603: 129 pgs: 129 active+clean; 194 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.7 MiB/s wr, 219 op/s 2024-05-08T20:19:32.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:31 smithi083 ceph-mon[38695]: pgmap v603: 129 pgs: 129 active+clean; 194 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.7 MiB/s wr, 219 op/s 2024-05-08T20:19:32.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1759774 16.04 MB/sec execute 476 sec latency 58.403 ms 2024-05-08T20:19:33.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 991305 8.95 MB/sec execute 477 sec latency 4.565 ms 2024-05-08T20:19:33.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1762595 16.04 MB/sec execute 477 sec latency 65.048 ms 2024-05-08T20:19:34.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 992641 8.94 MB/sec execute 478 sec latency 13.054 ms 2024-05-08T20:19:34.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:33 smithi188 ceph-mon[48761]: pgmap v604: 129 pgs: 129 active+clean; 194 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.3 MiB/s wr, 162 op/s 2024-05-08T20:19:34.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:33 smithi083 ceph-mon[38695]: pgmap v604: 129 pgs: 129 active+clean; 194 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.3 MiB/s wr, 162 op/s 2024-05-08T20:19:34.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1766054 16.04 MB/sec execute 478 sec latency 57.249 ms 2024-05-08T20:19:35.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 994520 8.94 MB/sec execute 479 sec latency 11.684 ms 2024-05-08T20:19:35.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1769531 16.05 MB/sec execute 479 sec latency 68.007 ms 2024-05-08T20:19:36.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 995793 8.94 MB/sec execute 480 sec latency 8.750 ms 2024-05-08T20:19:36.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:35 smithi188 ceph-mon[48761]: pgmap v605: 129 pgs: 129 active+clean; 193 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 3.6 MiB/s wr, 174 op/s 2024-05-08T20:19:36.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:35 smithi083 ceph-mon[38695]: pgmap v605: 129 pgs: 129 active+clean; 193 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 3.6 MiB/s wr, 174 op/s 2024-05-08T20:19:36.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1773020 16.06 MB/sec execute 480 sec latency 52.065 ms 2024-05-08T20:19:37.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 997518 8.94 MB/sec execute 481 sec latency 21.101 ms 2024-05-08T20:19:37.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1775536 16.05 MB/sec execute 481 sec latency 58.421 ms 2024-05-08T20:19:38.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 999008 8.94 MB/sec execute 482 sec latency 6.811 ms 2024-05-08T20:19:38.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:37 smithi188 ceph-mon[48761]: pgmap v606: 129 pgs: 129 active+clean; 196 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.3 MiB/s wr, 211 op/s 2024-05-08T20:19:38.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:37 smithi083 ceph-mon[38695]: pgmap v606: 129 pgs: 129 active+clean; 196 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.3 MiB/s wr, 211 op/s 2024-05-08T20:19:38.577 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1778268 16.04 MB/sec execute 482 sec latency 58.453 ms 2024-05-08T20:19:39.245 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1000665 8.93 MB/sec execute 483 sec latency 22.603 ms 2024-05-08T20:19:39.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:19:39.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:19:39.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1780920 16.05 MB/sec execute 483 sec latency 65.229 ms 2024-05-08T20:19:40.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1002264 8.94 MB/sec execute 484 sec latency 7.456 ms 2024-05-08T20:19:40.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:39 smithi188 ceph-mon[48761]: pgmap v607: 129 pgs: 129 active+clean; 198 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 3.5 MiB/s wr, 186 op/s 2024-05-08T20:19:40.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:39 smithi083 ceph-mon[38695]: pgmap v607: 129 pgs: 129 active+clean; 198 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 3.5 MiB/s wr, 186 op/s 2024-05-08T20:19:40.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1783790 16.04 MB/sec execute 484 sec latency 87.927 ms 2024-05-08T20:19:41.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1003754 8.93 MB/sec execute 485 sec latency 25.809 ms 2024-05-08T20:19:41.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1783867 16.01 MB/sec execute 485 sec latency 364.571 ms 2024-05-08T20:19:42.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1005528 8.94 MB/sec execute 486 sec latency 13.535 ms 2024-05-08T20:19:42.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:41 smithi188 ceph-mon[48761]: pgmap v608: 129 pgs: 129 active+clean; 198 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.4 MiB/s wr, 202 op/s 2024-05-08T20:19:42.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:41 smithi083 ceph-mon[38695]: pgmap v608: 129 pgs: 129 active+clean; 198 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.4 MiB/s wr, 202 op/s 2024-05-08T20:19:42.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1786282 16.01 MB/sec execute 486 sec latency 341.655 ms 2024-05-08T20:19:43.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1006911 8.93 MB/sec execute 487 sec latency 21.641 ms 2024-05-08T20:19:43.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1789827 16.01 MB/sec execute 487 sec latency 62.086 ms 2024-05-08T20:19:44.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1008724 8.93 MB/sec execute 488 sec latency 12.286 ms 2024-05-08T20:19:44.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:43 smithi188 ceph-mon[48761]: pgmap v609: 129 pgs: 129 active+clean; 201 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.4 MiB/s wr, 175 op/s 2024-05-08T20:19:44.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:43 smithi083 ceph-mon[38695]: pgmap v609: 129 pgs: 129 active+clean; 201 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.4 MiB/s wr, 175 op/s 2024-05-08T20:19:44.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1793070 16.02 MB/sec execute 488 sec latency 58.248 ms 2024-05-08T20:19:45.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1010052 8.93 MB/sec execute 489 sec latency 10.360 ms 2024-05-08T20:19:45.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1796534 16.02 MB/sec execute 489 sec latency 57.604 ms 2024-05-08T20:19:46.193 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:45 smithi083 ceph-mon[38695]: pgmap v610: 129 pgs: 129 active+clean; 197 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.8 MiB/s wr, 196 op/s 2024-05-08T20:19:46.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1011831 8.93 MB/sec execute 490 sec latency 20.324 ms 2024-05-08T20:19:46.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:45 smithi188 ceph-mon[48761]: pgmap v610: 129 pgs: 129 active+clean; 197 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.8 MiB/s wr, 196 op/s 2024-05-08T20:19:46.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1799582 16.02 MB/sec execute 490 sec latency 50.082 ms 2024-05-08T20:19:47.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1013244 8.93 MB/sec execute 491 sec latency 10.644 ms 2024-05-08T20:19:47.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1802597 16.02 MB/sec execute 491 sec latency 56.673 ms 2024-05-08T20:19:48.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1014874 8.92 MB/sec execute 492 sec latency 22.932 ms 2024-05-08T20:19:48.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:48 smithi188 ceph-mon[48761]: pgmap v611: 129 pgs: 129 active+clean; 194 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.4 MiB/s wr, 198 op/s 2024-05-08T20:19:48.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:48 smithi083 ceph-mon[38695]: pgmap v611: 129 pgs: 129 active+clean; 194 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.4 MiB/s wr, 198 op/s 2024-05-08T20:19:48.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1805181 16.02 MB/sec execute 492 sec latency 56.785 ms 2024-05-08T20:19:49.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1016460 8.93 MB/sec execute 493 sec latency 14.867 ms 2024-05-08T20:19:49.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1808603 16.02 MB/sec execute 493 sec latency 56.810 ms 2024-05-08T20:19:50.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1017944 8.92 MB/sec execute 494 sec latency 18.590 ms 2024-05-08T20:19:50.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:50 smithi188 ceph-mon[48761]: pgmap v612: 129 pgs: 129 active+clean; 197 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.7 MiB/s wr, 199 op/s 2024-05-08T20:19:50.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:50 smithi083 ceph-mon[38695]: pgmap v612: 129 pgs: 129 active+clean; 197 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.7 MiB/s wr, 199 op/s 2024-05-08T20:19:50.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1812082 16.03 MB/sec execute 494 sec latency 56.395 ms 2024-05-08T20:19:51.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1019714 8.93 MB/sec execute 495 sec latency 6.486 ms 2024-05-08T20:19:51.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1815569 16.03 MB/sec execute 495 sec latency 58.354 ms 2024-05-08T20:19:52.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1021048 8.92 MB/sec execute 496 sec latency 35.501 ms 2024-05-08T20:19:52.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:52 smithi188 ceph-mon[48761]: pgmap v613: 129 pgs: 129 active+clean; 196 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.5 MiB/s wr, 193 op/s 2024-05-08T20:19:52.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:52 smithi083 ceph-mon[38695]: pgmap v613: 129 pgs: 129 active+clean; 196 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.5 MiB/s wr, 193 op/s 2024-05-08T20:19:52.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1817654 16.02 MB/sec execute 496 sec latency 119.485 ms 2024-05-08T20:19:53.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1022884 8.93 MB/sec execute 497 sec latency 5.563 ms 2024-05-08T20:19:53.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1820327 16.02 MB/sec execute 497 sec latency 62.685 ms 2024-05-08T20:19:54.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1024080 8.92 MB/sec execute 498 sec latency 12.343 ms 2024-05-08T20:19:54.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:54 smithi188 ceph-mon[48761]: pgmap v614: 129 pgs: 129 active+clean; 196 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.4 MiB/s wr, 170 op/s 2024-05-08T20:19:54.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:19:54.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:54 smithi083 ceph-mon[38695]: pgmap v614: 129 pgs: 129 active+clean; 196 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.4 MiB/s wr, 170 op/s 2024-05-08T20:19:54.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:19:54.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1822865 16.02 MB/sec execute 498 sec latency 75.153 ms 2024-05-08T20:19:55.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1025697 8.91 MB/sec execute 499 sec latency 11.857 ms 2024-05-08T20:19:55.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1826327 16.02 MB/sec execute 499 sec latency 57.914 ms 2024-05-08T20:19:56.194 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:56 smithi083 ceph-mon[38695]: pgmap v615: 129 pgs: 129 active+clean; 198 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.5 MiB/s wr, 182 op/s 2024-05-08T20:19:56.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1027191 8.92 MB/sec execute 500 sec latency 7.900 ms 2024-05-08T20:19:56.259 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:56 smithi188 ceph-mon[48761]: pgmap v615: 129 pgs: 129 active+clean; 198 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.5 MiB/s wr, 182 op/s 2024-05-08T20:19:56.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1829758 16.03 MB/sec execute 500 sec latency 66.628 ms 2024-05-08T20:19:57.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1028866 8.91 MB/sec execute 501 sec latency 9.021 ms 2024-05-08T20:19:57.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1832372 16.02 MB/sec execute 501 sec latency 60.223 ms 2024-05-08T20:19:58.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1030587 8.92 MB/sec execute 502 sec latency 10.417 ms 2024-05-08T20:19:58.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:19:58 smithi188 ceph-mon[48761]: pgmap v616: 129 pgs: 129 active+clean; 194 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.0 MiB/s wr, 175 op/s 2024-05-08T20:19:58.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:19:58 smithi083 ceph-mon[38695]: pgmap v616: 129 pgs: 129 active+clean; 194 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.0 MiB/s wr, 175 op/s 2024-05-08T20:19:58.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1835264 16.05 MB/sec execute 502 sec latency 66.574 ms 2024-05-08T20:19:59.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1032107 8.91 MB/sec execute 503 sec latency 13.303 ms 2024-05-08T20:19:59.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1838228 16.05 MB/sec execute 503 sec latency 58.118 ms 2024-05-08T20:20:00.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1033926 8.92 MB/sec execute 504 sec latency 4.860 ms 2024-05-08T20:20:00.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:00 smithi188 ceph-mon[48761]: pgmap v617: 129 pgs: 129 active+clean; 198 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 3.7 MiB/s wr, 209 op/s 2024-05-08T20:20:00.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:00 smithi188 ceph-mon[48761]: overall HEALTH_OK 2024-05-08T20:20:00.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:00 smithi083 ceph-mon[38695]: pgmap v617: 129 pgs: 129 active+clean; 198 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 3.7 MiB/s wr, 209 op/s 2024-05-08T20:20:00.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:00 smithi083 ceph-mon[38695]: overall HEALTH_OK 2024-05-08T20:20:00.579 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1841361 16.06 MB/sec execute 504 sec latency 62.986 ms 2024-05-08T20:20:01.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1035339 8.92 MB/sec execute 505 sec latency 16.142 ms 2024-05-08T20:20:01.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1844830 16.06 MB/sec execute 505 sec latency 62.349 ms 2024-05-08T20:20:02.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1037108 8.92 MB/sec execute 506 sec latency 13.085 ms 2024-05-08T20:20:02.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:02 smithi188 ceph-mon[48761]: pgmap v618: 129 pgs: 129 active+clean; 194 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.2 MiB/s wr, 178 op/s 2024-05-08T20:20:02.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:02 smithi083 ceph-mon[38695]: pgmap v618: 129 pgs: 129 active+clean; 194 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.2 MiB/s wr, 178 op/s 2024-05-08T20:20:02.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1848313 16.07 MB/sec execute 506 sec latency 58.529 ms 2024-05-08T20:20:03.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1038528 8.91 MB/sec execute 507 sec latency 11.906 ms 2024-05-08T20:20:03.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1851808 16.07 MB/sec execute 507 sec latency 58.405 ms 2024-05-08T20:20:04.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1040507 8.92 MB/sec execute 508 sec latency 9.422 ms 2024-05-08T20:20:04.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:04 smithi188 ceph-mon[48761]: pgmap v619: 129 pgs: 129 active+clean; 193 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.2 MiB/s wr, 160 op/s 2024-05-08T20:20:04.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:04 smithi083 ceph-mon[38695]: pgmap v619: 129 pgs: 129 active+clean; 193 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.2 MiB/s wr, 160 op/s 2024-05-08T20:20:04.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1855377 16.08 MB/sec execute 508 sec latency 57.823 ms 2024-05-08T20:20:05.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1041875 8.91 MB/sec execute 509 sec latency 9.107 ms 2024-05-08T20:20:05.578 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1857896 16.07 MB/sec execute 509 sec latency 51.766 ms 2024-05-08T20:20:06.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1043743 8.92 MB/sec execute 510 sec latency 19.350 ms 2024-05-08T20:20:06.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:06 smithi188 ceph-mon[48761]: pgmap v620: 129 pgs: 129 active+clean; 197 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 3.9 MiB/s wr, 213 op/s 2024-05-08T20:20:06.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:06 smithi083 ceph-mon[38695]: pgmap v620: 129 pgs: 129 active+clean; 197 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 3.9 MiB/s wr, 213 op/s 2024-05-08T20:20:06.579 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1860729 16.07 MB/sec execute 510 sec latency 51.097 ms 2024-05-08T20:20:07.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1045223 8.92 MB/sec execute 511 sec latency 15.907 ms 2024-05-08T20:20:07.579 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1863433 16.07 MB/sec execute 511 sec latency 61.021 ms 2024-05-08T20:20:08.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1046894 8.91 MB/sec execute 512 sec latency 9.034 ms 2024-05-08T20:20:08.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:08 smithi188 ceph-mon[48761]: pgmap v621: 129 pgs: 129 active+clean; 195 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.3 MiB/s wr, 184 op/s 2024-05-08T20:20:08.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:08 smithi083 ceph-mon[38695]: pgmap v621: 129 pgs: 129 active+clean; 195 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.3 MiB/s wr, 184 op/s 2024-05-08T20:20:08.579 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1866264 16.07 MB/sec execute 512 sec latency 54.272 ms 2024-05-08T20:20:09.246 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1048490 8.92 MB/sec execute 513 sec latency 13.239 ms 2024-05-08T20:20:09.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:20:09.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:20:09.579 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1869686 16.07 MB/sec execute 513 sec latency 56.429 ms 2024-05-08T20:20:10.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1050065 8.91 MB/sec execute 514 sec latency 13.733 ms 2024-05-08T20:20:10.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:10 smithi188 ceph-mon[48761]: pgmap v622: 129 pgs: 129 active+clean; 195 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 3.6 MiB/s wr, 182 op/s 2024-05-08T20:20:10.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:10 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:20:10.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:10 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:20:10.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:10 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:20:10.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:10 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:20:10.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:10 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:20:10.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:10 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:20:10.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:10 smithi083 ceph-mon[38695]: pgmap v622: 129 pgs: 129 active+clean; 195 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 3.6 MiB/s wr, 182 op/s 2024-05-08T20:20:10.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:10 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:20:10.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:10 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:20:10.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:10 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:20:10.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:10 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:20:10.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:10 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:20:10.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:10 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:20:10.579 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1873166 16.08 MB/sec execute 514 sec latency 47.400 ms 2024-05-08T20:20:11.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1051805 8.92 MB/sec execute 515 sec latency 11.593 ms 2024-05-08T20:20:11.579 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1876655 16.08 MB/sec execute 515 sec latency 48.628 ms 2024-05-08T20:20:12.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1053282 8.91 MB/sec execute 516 sec latency 20.716 ms 2024-05-08T20:20:12.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:12 smithi188 ceph-mon[48761]: pgmap v623: 129 pgs: 129 active+clean; 197 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.4 MiB/s wr, 196 op/s 2024-05-08T20:20:12.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:12 smithi083 ceph-mon[38695]: pgmap v623: 129 pgs: 129 active+clean; 197 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.4 MiB/s wr, 196 op/s 2024-05-08T20:20:12.579 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1879301 16.08 MB/sec execute 516 sec latency 58.402 ms 2024-05-08T20:20:13.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1055074 8.92 MB/sec execute 517 sec latency 10.082 ms 2024-05-08T20:20:13.579 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1880258 16.06 MB/sec execute 517 sec latency 533.317 ms 2024-05-08T20:20:14.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1056481 8.91 MB/sec execute 518 sec latency 18.732 ms 2024-05-08T20:20:14.343 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:14 smithi188 ceph-mon[48761]: pgmap v624: 129 pgs: 129 active+clean; 195 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.5 MiB/s wr, 177 op/s 2024-05-08T20:20:14.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:14 smithi083 ceph-mon[38695]: pgmap v624: 129 pgs: 129 active+clean; 195 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.5 MiB/s wr, 177 op/s 2024-05-08T20:20:14.579 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1881940 16.04 MB/sec execute 518 sec latency 400.487 ms 2024-05-08T20:20:15.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1058350 8.92 MB/sec execute 519 sec latency 7.682 ms 2024-05-08T20:20:15.579 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1884706 16.04 MB/sec execute 519 sec latency 60.844 ms 2024-05-08T20:20:16.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1059671 8.91 MB/sec execute 520 sec latency 17.807 ms 2024-05-08T20:20:16.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:16 smithi188 ceph-mon[48761]: pgmap v625: 129 pgs: 129 active+clean; 200 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.1 MiB/s wr, 211 op/s 2024-05-08T20:20:16.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:16 smithi083 ceph-mon[38695]: pgmap v625: 129 pgs: 129 active+clean; 200 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.1 MiB/s wr, 211 op/s 2024-05-08T20:20:16.579 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1887602 16.04 MB/sec execute 520 sec latency 52.051 ms 2024-05-08T20:20:17.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1061467 8.91 MB/sec execute 521 sec latency 10.070 ms 2024-05-08T20:20:17.579 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1890966 16.05 MB/sec execute 521 sec latency 58.211 ms 2024-05-08T20:20:18.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1062831 8.91 MB/sec execute 522 sec latency 18.122 ms 2024-05-08T20:20:18.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:18 smithi188 ceph-mon[48761]: pgmap v626: 129 pgs: 129 active+clean; 195 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.1 MiB/s wr, 175 op/s 2024-05-08T20:20:18.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:18 smithi083 ceph-mon[38695]: pgmap v626: 129 pgs: 129 active+clean; 195 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.1 MiB/s wr, 175 op/s 2024-05-08T20:20:18.579 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1894444 16.05 MB/sec execute 522 sec latency 59.845 ms 2024-05-08T20:20:19.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1064442 8.91 MB/sec execute 523 sec latency 25.505 ms 2024-05-08T20:20:19.579 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1897924 16.06 MB/sec execute 523 sec latency 61.866 ms 2024-05-08T20:20:20.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1066018 8.91 MB/sec execute 524 sec latency 10.296 ms 2024-05-08T20:20:20.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:20 smithi188 ceph-mon[48761]: pgmap v627: 129 pgs: 129 active+clean; 196 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 3.6 MiB/s wr, 200 op/s 2024-05-08T20:20:20.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:20 smithi083 ceph-mon[38695]: pgmap v627: 129 pgs: 129 active+clean; 196 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 3.6 MiB/s wr, 200 op/s 2024-05-08T20:20:20.580 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1900244 16.05 MB/sec execute 524 sec latency 64.052 ms 2024-05-08T20:20:21.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1067401 8.91 MB/sec execute 525 sec latency 13.107 ms 2024-05-08T20:20:21.580 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1903173 16.05 MB/sec execute 525 sec latency 55.887 ms 2024-05-08T20:20:22.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1069062 8.91 MB/sec execute 526 sec latency 12.667 ms 2024-05-08T20:20:22.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:22 smithi188 ceph-mon[48761]: pgmap v628: 129 pgs: 129 active+clean; 194 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.6 MiB/s wr, 211 op/s 2024-05-08T20:20:22.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:22 smithi083 ceph-mon[38695]: pgmap v628: 129 pgs: 129 active+clean; 194 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.6 MiB/s wr, 211 op/s 2024-05-08T20:20:22.580 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1905827 16.05 MB/sec execute 526 sec latency 65.489 ms 2024-05-08T20:20:23.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1070374 8.90 MB/sec execute 527 sec latency 14.087 ms 2024-05-08T20:20:23.580 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1908729 16.04 MB/sec execute 527 sec latency 58.103 ms 2024-05-08T20:20:24.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1072132 8.90 MB/sec execute 528 sec latency 7.802 ms 2024-05-08T20:20:24.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:24 smithi188 ceph-mon[48761]: pgmap v629: 129 pgs: 129 active+clean; 194 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.3 MiB/s wr, 147 op/s 2024-05-08T20:20:24.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:24 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:20:24.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:24 smithi083 ceph-mon[38695]: pgmap v629: 129 pgs: 129 active+clean; 194 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.3 MiB/s wr, 147 op/s 2024-05-08T20:20:24.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:20:24.580 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1912200 16.05 MB/sec execute 528 sec latency 165.597 ms 2024-05-08T20:20:25.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1073457 8.90 MB/sec execute 529 sec latency 13.523 ms 2024-05-08T20:20:25.580 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1915690 16.06 MB/sec execute 529 sec latency 58.406 ms 2024-05-08T20:20:26.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1075090 8.90 MB/sec execute 530 sec latency 12.646 ms 2024-05-08T20:20:26.301 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:26 smithi083 ceph-mon[38695]: pgmap v630: 129 pgs: 129 active+clean; 195 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.9 MiB/s wr, 199 op/s 2024-05-08T20:20:26.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:26 smithi188 ceph-mon[48761]: pgmap v630: 129 pgs: 129 active+clean; 195 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.9 MiB/s wr, 199 op/s 2024-05-08T20:20:26.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1918305 16.05 MB/sec execute 530 sec latency 55.104 ms 2024-05-08T20:20:27.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1076665 8.90 MB/sec execute 531 sec latency 14.005 ms 2024-05-08T20:20:27.580 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1920899 16.04 MB/sec execute 531 sec latency 60.213 ms 2024-05-08T20:20:28.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1078086 8.90 MB/sec execute 532 sec latency 7.399 ms 2024-05-08T20:20:28.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:28 smithi188 ceph-mon[48761]: pgmap v631: 129 pgs: 129 active+clean; 196 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.0 MiB/s wr, 176 op/s 2024-05-08T20:20:28.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:28 smithi083 ceph-mon[38695]: pgmap v631: 129 pgs: 129 active+clean; 196 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.0 MiB/s wr, 176 op/s 2024-05-08T20:20:28.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1923740 16.05 MB/sec execute 532 sec latency 52.037 ms 2024-05-08T20:20:29.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1079723 8.90 MB/sec execute 533 sec latency 10.905 ms 2024-05-08T20:20:29.580 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1926387 16.04 MB/sec execute 533 sec latency 214.714 ms 2024-05-08T20:20:30.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1081082 8.89 MB/sec execute 534 sec latency 17.960 ms 2024-05-08T20:20:30.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:30 smithi188 ceph-mon[48761]: pgmap v632: 129 pgs: 129 active+clean; 197 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.5 MiB/s wr, 178 op/s 2024-05-08T20:20:30.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:30 smithi083 ceph-mon[38695]: pgmap v632: 129 pgs: 129 active+clean; 197 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.5 MiB/s wr, 178 op/s 2024-05-08T20:20:30.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1929878 16.05 MB/sec execute 534 sec latency 56.448 ms 2024-05-08T20:20:31.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1082861 8.90 MB/sec execute 535 sec latency 16.199 ms 2024-05-08T20:20:31.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1933414 16.05 MB/sec execute 535 sec latency 50.059 ms 2024-05-08T20:20:32.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1084211 8.89 MB/sec execute 536 sec latency 21.693 ms 2024-05-08T20:20:32.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:32 smithi188 ceph-mon[48761]: pgmap v633: 129 pgs: 129 active+clean; 199 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.4 MiB/s wr, 187 op/s 2024-05-08T20:20:32.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:32 smithi083 ceph-mon[38695]: pgmap v633: 129 pgs: 129 active+clean; 199 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.4 MiB/s wr, 187 op/s 2024-05-08T20:20:32.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1935733 16.05 MB/sec execute 536 sec latency 60.812 ms 2024-05-08T20:20:33.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1085850 8.89 MB/sec execute 537 sec latency 11.061 ms 2024-05-08T20:20:33.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1938664 16.04 MB/sec execute 537 sec latency 61.843 ms 2024-05-08T20:20:34.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1087420 8.89 MB/sec execute 538 sec latency 13.615 ms 2024-05-08T20:20:34.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:34 smithi188 ceph-mon[48761]: pgmap v634: 129 pgs: 129 active+clean; 199 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.3 MiB/s wr, 165 op/s 2024-05-08T20:20:34.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:34 smithi083 ceph-mon[38695]: pgmap v634: 129 pgs: 129 active+clean; 199 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.3 MiB/s wr, 165 op/s 2024-05-08T20:20:34.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1941466 16.05 MB/sec execute 538 sec latency 61.647 ms 2024-05-08T20:20:35.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1088985 8.89 MB/sec execute 539 sec latency 22.938 ms 2024-05-08T20:20:35.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1944366 16.04 MB/sec execute 539 sec latency 58.369 ms 2024-05-08T20:20:36.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1090736 8.90 MB/sec execute 540 sec latency 14.886 ms 2024-05-08T20:20:36.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:36 smithi188 ceph-mon[48761]: pgmap v635: 129 pgs: 129 active+clean; 195 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 3.6 MiB/s wr, 177 op/s 2024-05-08T20:20:36.427 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:36 smithi083 ceph-mon[38695]: pgmap v635: 129 pgs: 129 active+clean; 195 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 3.6 MiB/s wr, 177 op/s 2024-05-08T20:20:36.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1947732 16.05 MB/sec execute 540 sec latency 56.935 ms 2024-05-08T20:20:37.247 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1092068 8.89 MB/sec execute 541 sec latency 15.476 ms 2024-05-08T20:20:37.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1951207 16.05 MB/sec execute 541 sec latency 58.542 ms 2024-05-08T20:20:38.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1093830 8.89 MB/sec execute 542 sec latency 9.058 ms 2024-05-08T20:20:38.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:38 smithi188 ceph-mon[48761]: pgmap v636: 129 pgs: 129 active+clean; 196 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.9 MiB/s wr, 158 op/s 2024-05-08T20:20:38.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:38 smithi083 ceph-mon[38695]: pgmap v636: 129 pgs: 129 active+clean; 196 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.9 MiB/s wr, 158 op/s 2024-05-08T20:20:38.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1954695 16.06 MB/sec execute 542 sec latency 60.355 ms 2024-05-08T20:20:39.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1095175 8.89 MB/sec execute 543 sec latency 13.382 ms 2024-05-08T20:20:39.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:20:39.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:20:39.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1958240 16.06 MB/sec execute 543 sec latency 53.119 ms 2024-05-08T20:20:40.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1096946 8.89 MB/sec execute 544 sec latency 16.631 ms 2024-05-08T20:20:40.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:40 smithi188 ceph-mon[48761]: pgmap v637: 129 pgs: 129 active+clean; 200 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 3.6 MiB/s wr, 192 op/s 2024-05-08T20:20:40.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:40 smithi083 ceph-mon[38695]: pgmap v637: 129 pgs: 129 active+clean; 200 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 3.6 MiB/s wr, 192 op/s 2024-05-08T20:20:40.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1960863 16.06 MB/sec execute 544 sec latency 61.857 ms 2024-05-08T20:20:41.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1098295 8.89 MB/sec execute 545 sec latency 18.105 ms 2024-05-08T20:20:41.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1964102 16.06 MB/sec execute 545 sec latency 56.834 ms 2024-05-08T20:20:42.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1099928 8.88 MB/sec execute 546 sec latency 17.128 ms 2024-05-08T20:20:42.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:42 smithi188 ceph-mon[48761]: pgmap v638: 129 pgs: 129 active+clean; 200 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.4 MiB/s wr, 202 op/s 2024-05-08T20:20:42.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:42 smithi083 ceph-mon[38695]: pgmap v638: 129 pgs: 129 active+clean; 200 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.4 MiB/s wr, 202 op/s 2024-05-08T20:20:42.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1967419 16.07 MB/sec execute 546 sec latency 57.776 ms 2024-05-08T20:20:43.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1101502 8.89 MB/sec execute 547 sec latency 10.873 ms 2024-05-08T20:20:43.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1970614 16.06 MB/sec execute 547 sec latency 58.388 ms 2024-05-08T20:20:44.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1102920 8.88 MB/sec execute 548 sec latency 16.925 ms 2024-05-08T20:20:44.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:44 smithi188 ceph-mon[48761]: pgmap v639: 129 pgs: 129 active+clean; 198 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.3 MiB/s wr, 163 op/s 2024-05-08T20:20:44.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:44 smithi083 ceph-mon[38695]: pgmap v639: 129 pgs: 129 active+clean; 198 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.3 MiB/s wr, 163 op/s 2024-05-08T20:20:44.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1973637 16.07 MB/sec execute 548 sec latency 58.145 ms 2024-05-08T20:20:45.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1104635 8.89 MB/sec execute 549 sec latency 14.983 ms 2024-05-08T20:20:45.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1975972 16.06 MB/sec execute 549 sec latency 443.178 ms 2024-05-08T20:20:46.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1105872 8.88 MB/sec execute 550 sec latency 10.659 ms 2024-05-08T20:20:46.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:46 smithi188 ceph-mon[48761]: pgmap v640: 129 pgs: 129 active+clean; 197 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 3.5 MiB/s wr, 173 op/s 2024-05-08T20:20:46.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:46 smithi083 ceph-mon[38695]: pgmap v640: 129 pgs: 129 active+clean; 197 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 3.5 MiB/s wr, 173 op/s 2024-05-08T20:20:46.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1976211 16.03 MB/sec execute 550 sec latency 359.793 ms 2024-05-08T20:20:47.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1107680 8.88 MB/sec execute 551 sec latency 4.764 ms 2024-05-08T20:20:47.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1979636 16.04 MB/sec execute 551 sec latency 49.473 ms 2024-05-08T20:20:48.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1109061 8.88 MB/sec execute 552 sec latency 20.123 ms 2024-05-08T20:20:48.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:48 smithi188 ceph-mon[48761]: pgmap v641: 129 pgs: 129 active+clean; 196 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.2 MiB/s wr, 212 op/s 2024-05-08T20:20:48.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:48 smithi083 ceph-mon[38695]: pgmap v641: 129 pgs: 129 active+clean; 196 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.2 MiB/s wr, 212 op/s 2024-05-08T20:20:48.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1983065 16.04 MB/sec execute 552 sec latency 58.104 ms 2024-05-08T20:20:49.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1110671 8.87 MB/sec execute 553 sec latency 20.515 ms 2024-05-08T20:20:49.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1985673 16.04 MB/sec execute 553 sec latency 66.779 ms 2024-05-08T20:20:50.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1112238 8.88 MB/sec execute 554 sec latency 14.180 ms 2024-05-08T20:20:50.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:50 smithi188 ceph-mon[48761]: pgmap v642: 129 pgs: 129 active+clean; 196 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 3.5 MiB/s wr, 192 op/s 2024-05-08T20:20:50.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:50 smithi083 ceph-mon[38695]: pgmap v642: 129 pgs: 129 active+clean; 196 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 3.5 MiB/s wr, 192 op/s 2024-05-08T20:20:50.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1988691 16.04 MB/sec execute 554 sec latency 56.939 ms 2024-05-08T20:20:51.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1113796 8.87 MB/sec execute 555 sec latency 26.717 ms 2024-05-08T20:20:51.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1991361 16.04 MB/sec execute 555 sec latency 50.193 ms 2024-05-08T20:20:52.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1115550 8.88 MB/sec execute 556 sec latency 16.316 ms 2024-05-08T20:20:52.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:52 smithi188 ceph-mon[48761]: pgmap v643: 129 pgs: 129 active+clean; 199 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.3 MiB/s wr, 190 op/s 2024-05-08T20:20:52.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:52 smithi083 ceph-mon[38695]: pgmap v643: 129 pgs: 129 active+clean; 199 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.3 MiB/s wr, 190 op/s 2024-05-08T20:20:52.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1994005 16.03 MB/sec execute 556 sec latency 50.093 ms 2024-05-08T20:20:53.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1116872 8.87 MB/sec execute 557 sec latency 11.392 ms 2024-05-08T20:20:53.582 INFO:tasks.workunit.client.0.smithi083.stdout: 1 1997369 16.04 MB/sec execute 557 sec latency 47.819 ms 2024-05-08T20:20:54.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1118648 8.87 MB/sec execute 558 sec latency 16.152 ms 2024-05-08T20:20:54.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:54 smithi188 ceph-mon[48761]: pgmap v644: 129 pgs: 129 active+clean; 199 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.3 MiB/s wr, 162 op/s 2024-05-08T20:20:54.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:20:54.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:54 smithi083 ceph-mon[38695]: pgmap v644: 129 pgs: 129 active+clean; 199 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.3 MiB/s wr, 162 op/s 2024-05-08T20:20:54.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:20:54.581 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2000848 16.04 MB/sec execute 558 sec latency 75.092 ms 2024-05-08T20:20:55.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1120094 8.87 MB/sec execute 559 sec latency 15.057 ms 2024-05-08T20:20:55.582 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2004332 16.05 MB/sec execute 559 sec latency 49.071 ms 2024-05-08T20:20:56.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1122085 8.88 MB/sec execute 560 sec latency 15.432 ms 2024-05-08T20:20:56.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:56 smithi188 ceph-mon[48761]: pgmap v645: 129 pgs: 129 active+clean; 201 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 3.7 MiB/s wr, 194 op/s 2024-05-08T20:20:56.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:56 smithi083 ceph-mon[38695]: pgmap v645: 129 pgs: 129 active+clean; 201 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 3.7 MiB/s wr, 194 op/s 2024-05-08T20:20:56.582 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2006953 16.05 MB/sec execute 560 sec latency 55.439 ms 2024-05-08T20:20:57.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1123422 8.87 MB/sec execute 561 sec latency 9.193 ms 2024-05-08T20:20:57.582 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2009090 16.04 MB/sec execute 561 sec latency 92.820 ms 2024-05-08T20:20:58.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1125175 8.87 MB/sec execute 562 sec latency 16.572 ms 2024-05-08T20:20:58.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:20:58 smithi188 ceph-mon[48761]: pgmap v646: 129 pgs: 129 active+clean; 203 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.4 MiB/s wr, 204 op/s 2024-05-08T20:20:58.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:20:58 smithi083 ceph-mon[38695]: pgmap v646: 129 pgs: 129 active+clean; 203 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.4 MiB/s wr, 204 op/s 2024-05-08T20:20:58.582 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2011731 16.03 MB/sec execute 562 sec latency 58.711 ms 2024-05-08T20:20:59.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1126724 8.87 MB/sec execute 563 sec latency 17.141 ms 2024-05-08T20:20:59.582 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2015098 16.04 MB/sec execute 563 sec latency 55.019 ms 2024-05-08T20:21:00.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1128409 8.87 MB/sec execute 564 sec latency 15.260 ms 2024-05-08T20:21:00.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:00 smithi188 ceph-mon[48761]: pgmap v647: 129 pgs: 129 active+clean; 200 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 3.7 MiB/s wr, 169 op/s 2024-05-08T20:21:00.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:00 smithi083 ceph-mon[38695]: pgmap v647: 129 pgs: 129 active+clean; 200 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 3.7 MiB/s wr, 169 op/s 2024-05-08T20:21:00.582 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2018566 16.04 MB/sec execute 564 sec latency 110.355 ms 2024-05-08T20:21:01.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1130042 8.87 MB/sec execute 565 sec latency 7.000 ms 2024-05-08T20:21:01.582 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2022063 16.05 MB/sec execute 565 sec latency 56.016 ms 2024-05-08T20:21:02.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1131703 8.87 MB/sec execute 566 sec latency 21.148 ms 2024-05-08T20:21:02.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:02 smithi188 ceph-mon[48761]: pgmap v648: 129 pgs: 129 active+clean; 197 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.6 MiB/s wr, 215 op/s 2024-05-08T20:21:02.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:02 smithi083 ceph-mon[38695]: pgmap v648: 129 pgs: 129 active+clean; 197 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.6 MiB/s wr, 215 op/s 2024-05-08T20:21:02.582 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2024148 16.04 MB/sec execute 566 sec latency 81.348 ms 2024-05-08T20:21:03.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1133421 8.88 MB/sec execute 567 sec latency 13.778 ms 2024-05-08T20:21:03.582 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2026813 16.03 MB/sec execute 567 sec latency 66.782 ms 2024-05-08T20:21:04.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1134917 8.87 MB/sec execute 568 sec latency 15.891 ms 2024-05-08T20:21:04.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:04 smithi188 ceph-mon[48761]: pgmap v649: 129 pgs: 129 active+clean; 197 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.4 MiB/s wr, 171 op/s 2024-05-08T20:21:04.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:04 smithi083 ceph-mon[38695]: pgmap v649: 129 pgs: 129 active+clean; 197 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.4 MiB/s wr, 171 op/s 2024-05-08T20:21:04.582 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2029399 16.03 MB/sec execute 568 sec latency 54.097 ms 2024-05-08T20:21:05.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1136675 8.88 MB/sec execute 569 sec latency 13.573 ms 2024-05-08T20:21:05.582 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2032827 16.04 MB/sec execute 569 sec latency 53.480 ms 2024-05-08T20:21:06.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1138088 8.87 MB/sec execute 570 sec latency 32.488 ms 2024-05-08T20:21:06.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:06 smithi188 ceph-mon[48761]: pgmap v650: 129 pgs: 129 active+clean; 198 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 3.9 MiB/s wr, 203 op/s 2024-05-08T20:21:06.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:06 smithi083 ceph-mon[38695]: pgmap v650: 129 pgs: 129 active+clean; 198 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 3.9 MiB/s wr, 203 op/s 2024-05-08T20:21:06.582 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2036303 16.04 MB/sec execute 570 sec latency 54.092 ms 2024-05-08T20:21:07.248 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1139945 8.87 MB/sec execute 571 sec latency 17.264 ms 2024-05-08T20:21:07.582 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2039789 16.05 MB/sec execute 571 sec latency 56.569 ms 2024-05-08T20:21:08.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1141229 8.87 MB/sec execute 572 sec latency 13.670 ms 2024-05-08T20:21:08.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:08 smithi188 ceph-mon[48761]: pgmap v651: 129 pgs: 129 active+clean; 200 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.2 MiB/s wr, 176 op/s 2024-05-08T20:21:08.443 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:08 smithi083 ceph-mon[38695]: pgmap v651: 129 pgs: 129 active+clean; 200 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.2 MiB/s wr, 176 op/s 2024-05-08T20:21:08.582 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2042298 16.04 MB/sec execute 572 sec latency 60.415 ms 2024-05-08T20:21:09.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1142810 8.87 MB/sec execute 573 sec latency 16.566 ms 2024-05-08T20:21:09.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:21:09.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:21:09.582 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2045133 16.04 MB/sec execute 573 sec latency 61.398 ms 2024-05-08T20:21:10.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1144314 8.87 MB/sec execute 574 sec latency 12.701 ms 2024-05-08T20:21:10.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:10 smithi188 ceph-mon[48761]: pgmap v652: 129 pgs: 129 active+clean; 202 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 3.6 MiB/s wr, 199 op/s 2024-05-08T20:21:10.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:10 smithi083 ceph-mon[38695]: pgmap v652: 129 pgs: 129 active+clean; 202 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 3.6 MiB/s wr, 199 op/s 2024-05-08T20:21:10.582 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2047834 16.04 MB/sec execute 574 sec latency 60.160 ms 2024-05-08T20:21:11.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1145828 8.86 MB/sec execute 575 sec latency 16.840 ms 2024-05-08T20:21:11.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:11 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:21:11.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:11 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:21:11.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:11 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:21:11.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:11 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:21:11.582 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2050685 16.04 MB/sec execute 575 sec latency 57.925 ms 2024-05-08T20:21:11.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:11 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:21:11.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:11 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:21:11.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:11 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:21:11.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:11 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:21:12.194 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:12 smithi083 ceph-mon[38695]: pgmap v653: 129 pgs: 129 active+clean; 202 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.4 MiB/s wr, 213 op/s 2024-05-08T20:21:12.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1147526 8.87 MB/sec execute 576 sec latency 8.956 ms 2024-05-08T20:21:12.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:12 smithi188 ceph-mon[48761]: pgmap v653: 129 pgs: 129 active+clean; 202 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.4 MiB/s wr, 213 op/s 2024-05-08T20:21:12.586 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2054142 16.04 MB/sec execute 576 sec latency 56.676 ms 2024-05-08T20:21:13.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1148851 8.86 MB/sec execute 577 sec latency 20.274 ms 2024-05-08T20:21:13.582 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2057624 16.05 MB/sec execute 577 sec latency 49.471 ms 2024-05-08T20:21:14.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1150607 8.87 MB/sec execute 578 sec latency 9.427 ms 2024-05-08T20:21:14.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:14 smithi083 ceph-mon[38695]: pgmap v654: 129 pgs: 129 active+clean; 203 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.4 MiB/s wr, 154 op/s 2024-05-08T20:21:14.582 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2061110 16.05 MB/sec execute 578 sec latency 56.723 ms 2024-05-08T20:21:14.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:14 smithi188 ceph-mon[48761]: pgmap v654: 129 pgs: 129 active+clean; 203 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.4 MiB/s wr, 154 op/s 2024-05-08T20:21:15.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1151904 8.86 MB/sec execute 579 sec latency 13.358 ms 2024-05-08T20:21:15.582 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2063411 16.04 MB/sec execute 579 sec latency 57.561 ms 2024-05-08T20:21:16.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1153751 8.86 MB/sec execute 580 sec latency 8.722 ms 2024-05-08T20:21:16.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:16 smithi083 ceph-mon[38695]: pgmap v655: 129 pgs: 129 active+clean; 200 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.7 MiB/s wr, 202 op/s 2024-05-08T20:21:16.582 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2066327 16.04 MB/sec execute 580 sec latency 66.815 ms 2024-05-08T20:21:16.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:16 smithi188 ceph-mon[48761]: pgmap v655: 129 pgs: 129 active+clean; 200 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.7 MiB/s wr, 202 op/s 2024-05-08T20:21:17.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1155218 8.86 MB/sec execute 581 sec latency 12.484 ms 2024-05-08T20:21:17.583 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2068326 16.03 MB/sec execute 581 sec latency 221.808 ms 2024-05-08T20:21:18.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1156881 8.86 MB/sec execute 582 sec latency 16.879 ms 2024-05-08T20:21:18.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:18 smithi083 ceph-mon[38695]: pgmap v656: 129 pgs: 129 active+clean; 201 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.1 MiB/s wr, 183 op/s 2024-05-08T20:21:18.582 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2068331 16.01 MB/sec execute 582 sec latency 475.316 ms 2024-05-08T20:21:18.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:18 smithi188 ceph-mon[48761]: pgmap v656: 129 pgs: 129 active+clean; 201 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.1 MiB/s wr, 183 op/s 2024-05-08T20:21:19.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1158520 8.86 MB/sec execute 583 sec latency 6.125 ms 2024-05-08T20:21:19.583 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2071807 16.01 MB/sec execute 583 sec latency 433.526 ms 2024-05-08T20:21:20.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1160227 8.86 MB/sec execute 584 sec latency 22.661 ms 2024-05-08T20:21:20.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:20 smithi083 ceph-mon[38695]: pgmap v657: 129 pgs: 129 active+clean; 197 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 3.4 MiB/s wr, 173 op/s 2024-05-08T20:21:20.583 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2075303 16.02 MB/sec execute 584 sec latency 58.445 ms 2024-05-08T20:21:20.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:20 smithi188 ceph-mon[48761]: pgmap v657: 129 pgs: 129 active+clean; 197 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 3.4 MiB/s wr, 173 op/s 2024-05-08T20:21:21.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1161875 8.87 MB/sec execute 585 sec latency 14.016 ms 2024-05-08T20:21:21.583 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2078846 16.02 MB/sec execute 585 sec latency 58.450 ms 2024-05-08T20:21:22.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1163432 8.86 MB/sec execute 586 sec latency 15.889 ms 2024-05-08T20:21:22.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:22 smithi083 ceph-mon[38695]: pgmap v658: 129 pgs: 129 active+clean; 199 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.3 MiB/s wr, 195 op/s 2024-05-08T20:21:22.583 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2081489 16.02 MB/sec execute 586 sec latency 47.937 ms 2024-05-08T20:21:22.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:22 smithi188 ceph-mon[48761]: pgmap v658: 129 pgs: 129 active+clean; 199 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.3 MiB/s wr, 195 op/s 2024-05-08T20:21:23.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1165250 8.87 MB/sec execute 587 sec latency 18.336 ms 2024-05-08T20:21:23.583 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2084471 16.02 MB/sec execute 587 sec latency 74.862 ms 2024-05-08T20:21:24.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1166655 8.86 MB/sec execute 588 sec latency 20.381 ms 2024-05-08T20:21:24.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:24 smithi083 ceph-mon[38695]: pgmap v659: 129 pgs: 129 active+clean; 199 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.3 MiB/s wr, 167 op/s 2024-05-08T20:21:24.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:21:24.583 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2086896 16.01 MB/sec execute 588 sec latency 172.918 ms 2024-05-08T20:21:24.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:24 smithi188 ceph-mon[48761]: pgmap v659: 129 pgs: 129 active+clean; 199 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.3 MiB/s wr, 167 op/s 2024-05-08T20:21:24.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:24 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:21:25.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1168393 8.86 MB/sec execute 589 sec latency 17.730 ms 2024-05-08T20:21:25.583 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2089720 16.01 MB/sec execute 589 sec latency 57.863 ms 2024-05-08T20:21:26.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1169959 8.86 MB/sec execute 590 sec latency 11.418 ms 2024-05-08T20:21:26.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:26 smithi083 ceph-mon[38695]: pgmap v660: 129 pgs: 129 active+clean; 200 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 3.6 MiB/s wr, 178 op/s 2024-05-08T20:21:26.583 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2093153 16.02 MB/sec execute 590 sec latency 56.807 ms 2024-05-08T20:21:26.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:26 smithi188 ceph-mon[48761]: pgmap v660: 129 pgs: 129 active+clean; 200 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 3.6 MiB/s wr, 178 op/s 2024-05-08T20:21:27.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1171853 8.86 MB/sec execute 591 sec latency 12.459 ms 2024-05-08T20:21:27.583 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2096630 16.02 MB/sec execute 591 sec latency 53.531 ms 2024-05-08T20:21:28.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1173168 8.86 MB/sec execute 592 sec latency 13.312 ms 2024-05-08T20:21:28.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:28 smithi083 ceph-mon[38695]: pgmap v661: 129 pgs: 129 active+clean; 198 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.3 MiB/s wr, 185 op/s 2024-05-08T20:21:28.583 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2100155 16.03 MB/sec execute 592 sec latency 46.702 ms 2024-05-08T20:21:28.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:28 smithi188 ceph-mon[48761]: pgmap v661: 129 pgs: 129 active+clean; 198 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.3 MiB/s wr, 185 op/s 2024-05-08T20:21:29.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1174987 8.86 MB/sec execute 593 sec latency 8.572 ms 2024-05-08T20:21:29.583 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2102738 16.02 MB/sec execute 593 sec latency 97.772 ms 2024-05-08T20:21:30.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1176377 8.86 MB/sec execute 594 sec latency 11.264 ms 2024-05-08T20:21:30.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:30 smithi083 ceph-mon[38695]: pgmap v662: 129 pgs: 129 active+clean; 199 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 3.6 MiB/s wr, 182 op/s 2024-05-08T20:21:30.583 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2105745 16.02 MB/sec execute 594 sec latency 57.567 ms 2024-05-08T20:21:30.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:30 smithi188 ceph-mon[48761]: pgmap v662: 129 pgs: 129 active+clean; 199 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 3.6 MiB/s wr, 182 op/s 2024-05-08T20:21:31.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1178003 8.86 MB/sec execute 595 sec latency 17.770 ms 2024-05-08T20:21:31.583 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2108422 16.02 MB/sec execute 595 sec latency 52.111 ms 2024-05-08T20:21:32.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1179582 8.86 MB/sec execute 596 sec latency 12.243 ms 2024-05-08T20:21:32.391 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:32 smithi188 ceph-mon[48761]: pgmap v663: 129 pgs: 129 active+clean; 202 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.7 MiB/s wr, 223 op/s 2024-05-08T20:21:32.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:32 smithi083 ceph-mon[38695]: pgmap v663: 129 pgs: 129 active+clean; 202 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.7 MiB/s wr, 223 op/s 2024-05-08T20:21:32.583 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2111959 16.02 MB/sec execute 596 sec latency 50.811 ms 2024-05-08T20:21:33.250 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1181022 8.86 MB/sec execute 597 sec latency 22.416 ms 2024-05-08T20:21:33.583 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2114608 16.02 MB/sec execute 597 sec latency 58.789 ms 2024-05-08T20:21:34.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1182711 8.86 MB/sec execute 598 sec latency 18.734 ms 2024-05-08T20:21:34.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:34 smithi083 ceph-mon[38695]: pgmap v664: 129 pgs: 129 active+clean; 202 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.4 MiB/s wr, 157 op/s 2024-05-08T20:21:34.583 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2117966 16.03 MB/sec execute 598 sec latency 77.681 ms 2024-05-08T20:21:34.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:34 smithi188 ceph-mon[48761]: pgmap v664: 129 pgs: 129 active+clean; 202 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.4 MiB/s wr, 157 op/s 2024-05-08T20:21:35.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 1183980 8.85 MB/sec execute 599 sec latency 11.617 ms 2024-05-08T20:21:35.583 INFO:tasks.workunit.client.0.smithi083.stdout: 1 2121453 16.03 MB/sec execute 599 sec latency 52.551 ms 2024-05-08T20:21:36.249 INFO:tasks.workunit.client.1.smithi188.stdout: 1 cleanup 600 sec 2024-05-08T20:21:36.342 INFO:tasks.workunit.client.1.smithi188.stdout: 0 cleanup 600 sec 2024-05-08T20:21:36.342 INFO:tasks.workunit.client.1.smithi188.stdout: 2024-05-08T20:21:36.342 INFO:tasks.workunit.client.1.smithi188.stdout: Operation Count AvgLat MaxLat 2024-05-08T20:21:36.342 INFO:tasks.workunit.client.1.smithi188.stdout: ---------------------------------------- 2024-05-08T20:21:36.342 INFO:tasks.workunit.client.1.smithi188.stdout: NTCreateX 169339 1.140 25.173 2024-05-08T20:21:36.342 INFO:tasks.workunit.client.1.smithi188.stdout: Close 124411 0.017 3.552 2024-05-08T20:21:36.343 INFO:tasks.workunit.client.1.smithi188.stdout: Rename 7172 2.926 24.276 2024-05-08T20:21:36.343 INFO:tasks.workunit.client.1.smithi188.stdout: Unlink 34188 0.912 11.746 2024-05-08T20:21:36.343 INFO:tasks.workunit.client.1.smithi188.stdout: Deltree 4 41.490 77.771 2024-05-08T20:21:36.343 INFO:tasks.workunit.client.1.smithi188.stdout: Mkdir 2 0.149 0.151 2024-05-08T20:21:36.343 INFO:tasks.workunit.client.1.smithi188.stdout: Qpathinfo 153521 0.722 20.313 2024-05-08T20:21:36.343 INFO:tasks.workunit.client.1.smithi188.stdout: Qfileinfo 26901 0.049 1.796 2024-05-08T20:21:36.343 INFO:tasks.workunit.client.1.smithi188.stdout: Qfsinfo 28128 0.225 6.584 2024-05-08T20:21:36.343 INFO:tasks.workunit.client.1.smithi188.stdout: Sfileinfo 13808 0.366 6.606 2024-05-08T20:21:36.343 INFO:tasks.workunit.client.1.smithi188.stdout: Find 59334 3.279 35.489 2024-05-08T20:21:36.343 INFO:tasks.workunit.client.1.smithi188.stdout: WriteX 84389 0.099 4.818 2024-05-08T20:21:36.343 INFO:tasks.workunit.client.1.smithi188.stdout: ReadX 265410 0.003 1.319 2024-05-08T20:21:36.343 INFO:tasks.workunit.client.1.smithi188.stdout: LockX 552 0.194 2.797 2024-05-08T20:21:36.343 INFO:tasks.workunit.client.1.smithi188.stdout: UnlockX 552 0.136 0.353 2024-05-08T20:21:36.343 INFO:tasks.workunit.client.1.smithi188.stdout: Flush 11874 1.927 19.066 2024-05-08T20:21:36.343 INFO:tasks.workunit.client.1.smithi188.stdout: 2024-05-08T20:21:36.343 INFO:tasks.workunit.client.1.smithi188.stdout:Throughput 8.8534 MB/sec 1 clients 1 procs max_latency=59.378 ms 2024-05-08T20:21:36.344 INFO:tasks.workunit.client.1.smithi188.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-05-08T20:21:36.344 INFO:tasks.workunit.client.1.smithi188.stdout: 2024-05-08T20:21:36.344 INFO:tasks.workunit.client.1.smithi188.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-05-08T20:21:36.344 INFO:tasks.workunit.client.1.smithi188.stdout:8 of 10 processes prepared for launch 0 sec 2024-05-08T20:21:36.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:36 smithi083 ceph-mon[38695]: pgmap v665: 129 pgs: 129 active+clean; 200 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 3.9 MiB/s wr, 199 op/s 2024-05-08T20:21:36.444 INFO:tasks.workunit.client.1.smithi188.stdout:10 of 10 processes prepared for launch 0 sec 2024-05-08T20:21:36.444 INFO:tasks.workunit.client.1.smithi188.stdout:releasing clients 2024-05-08T20:21:36.583 INFO:tasks.workunit.client.0.smithi083.stdout: 1 cleanup 600 sec 2024-05-08T20:21:36.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:36 smithi188 ceph-mon[48761]: pgmap v665: 129 pgs: 129 active+clean; 200 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 3.9 MiB/s wr, 199 op/s 2024-05-08T20:21:36.603 INFO:tasks.workunit.client.0.smithi083.stdout: 0 cleanup 600 sec 2024-05-08T20:21:36.604 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-05-08T20:21:36.604 INFO:tasks.workunit.client.0.smithi083.stdout: Operation Count AvgLat MaxLat 2024-05-08T20:21:36.604 INFO:tasks.workunit.client.0.smithi083.stdout: ---------------------------------------- 2024-05-08T20:21:36.604 INFO:tasks.workunit.client.0.smithi083.stdout: NTCreateX 306260 0.009 3.797 2024-05-08T20:21:36.604 INFO:tasks.workunit.client.0.smithi083.stdout: Close 224963 0.001 0.046 2024-05-08T20:21:36.604 INFO:tasks.workunit.client.0.smithi083.stdout: Rename 12967 0.021 0.098 2024-05-08T20:21:36.604 INFO:tasks.workunit.client.0.smithi083.stdout: Unlink 61869 0.017 1.934 2024-05-08T20:21:36.604 INFO:tasks.workunit.client.0.smithi083.stdout: Deltree 8 1.730 3.555 2024-05-08T20:21:36.604 INFO:tasks.workunit.client.0.smithi083.stdout: Mkdir 4 0.002 0.002 2024-05-08T20:21:36.604 INFO:tasks.workunit.client.0.smithi083.stdout: Qpathinfo 277613 0.004 0.367 2024-05-08T20:21:36.604 INFO:tasks.workunit.client.0.smithi083.stdout: Qfileinfo 48653 0.001 0.029 2024-05-08T20:21:36.604 INFO:tasks.workunit.client.0.smithi083.stdout: Qfsinfo 50911 0.002 0.125 2024-05-08T20:21:36.604 INFO:tasks.workunit.client.0.smithi083.stdout: Sfileinfo 24935 0.009 76.921 2024-05-08T20:21:36.604 INFO:tasks.workunit.client.0.smithi083.stdout: Find 107340 0.016 0.120 2024-05-08T20:21:36.605 INFO:tasks.workunit.client.0.smithi083.stdout: WriteX 152744 0.013 4.254 2024-05-08T20:21:36.605 INFO:tasks.workunit.client.0.smithi083.stdout: ReadX 480168 0.002 0.504 2024-05-08T20:21:36.605 INFO:tasks.workunit.client.0.smithi083.stdout: LockX 998 0.002 0.019 2024-05-08T20:21:36.605 INFO:tasks.workunit.client.0.smithi083.stdout: UnlockX 998 0.001 0.011 2024-05-08T20:21:36.605 INFO:tasks.workunit.client.0.smithi083.stdout: Flush 21458 27.378 678.663 2024-05-08T20:21:36.605 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-05-08T20:21:36.605 INFO:tasks.workunit.client.0.smithi083.stdout:Throughput 16.0311 MB/sec 1 clients 1 procs max_latency=678.671 ms 2024-05-08T20:21:36.605 INFO:tasks.workunit.client.0.smithi083.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-05-08T20:21:36.605 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-05-08T20:21:36.605 INFO:tasks.workunit.client.0.smithi083.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-05-08T20:21:36.606 INFO:tasks.workunit.client.0.smithi083.stdout:5 of 10 processes prepared for launch 0 sec 2024-05-08T20:21:36.706 INFO:tasks.workunit.client.0.smithi083.stdout:10 of 10 processes prepared for launch 0 sec 2024-05-08T20:21:36.706 INFO:tasks.workunit.client.0.smithi083.stdout:releasing clients 2024-05-08T20:21:37.444 INFO:tasks.workunit.client.1.smithi188.stdout: 10 550 188.77 MB/sec warmup 1 sec latency 29.893 ms 2024-05-08T20:21:37.706 INFO:tasks.workunit.client.0.smithi083.stdout: 10 794 214.67 MB/sec warmup 1 sec latency 95.736 ms 2024-05-08T20:21:38.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:38 smithi083 ceph-mon[38695]: pgmap v666: 129 pgs: 129 active+clean; 199 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.3 MiB/s wr, 205 op/s 2024-05-08T20:21:38.444 INFO:tasks.workunit.client.1.smithi188.stdout: 10 1047 114.46 MB/sec warmup 2 sec latency 56.070 ms 2024-05-08T20:21:38.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:38 smithi188 ceph-mon[48761]: pgmap v666: 129 pgs: 129 active+clean; 199 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.3 MiB/s wr, 205 op/s 2024-05-08T20:21:38.706 INFO:tasks.workunit.client.0.smithi083.stdout: 10 1886 131.88 MB/sec warmup 2 sec latency 117.621 ms 2024-05-08T20:21:39.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:21:39.444 INFO:tasks.workunit.client.1.smithi188.stdout: 10 1620 87.22 MB/sec warmup 3 sec latency 59.627 ms 2024-05-08T20:21:39.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:21:39.706 INFO:tasks.workunit.client.0.smithi083.stdout: 10 3305 117.89 MB/sec warmup 3 sec latency 102.825 ms 2024-05-08T20:21:40.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:40 smithi083 ceph-mon[38695]: pgmap v667: 129 pgs: 129 active+clean; 254 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 8.5 MiB/s wr, 200 op/s 2024-05-08T20:21:40.444 INFO:tasks.workunit.client.1.smithi188.stdout: 10 2244 67.91 MB/sec warmup 4 sec latency 30.788 ms 2024-05-08T20:21:40.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:40 smithi188 ceph-mon[48761]: pgmap v667: 129 pgs: 129 active+clean; 254 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 8.5 MiB/s wr, 200 op/s 2024-05-08T20:21:40.706 INFO:tasks.workunit.client.0.smithi083.stdout: 10 4309 101.12 MB/sec warmup 4 sec latency 108.914 ms 2024-05-08T20:21:41.444 INFO:tasks.workunit.client.1.smithi188.stdout: 10 2931 66.58 MB/sec warmup 5 sec latency 61.482 ms 2024-05-08T20:21:41.706 INFO:tasks.workunit.client.0.smithi083.stdout: 10 5197 90.58 MB/sec warmup 5 sec latency 103.217 ms 2024-05-08T20:21:42.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:42 smithi083 ceph-mon[38695]: pgmap v668: 129 pgs: 129 active+clean; 381 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 21 MiB/s wr, 338 op/s 2024-05-08T20:21:42.444 INFO:tasks.workunit.client.1.smithi188.stdout: 10 3586 64.14 MB/sec warmup 6 sec latency 50.112 ms 2024-05-08T20:21:42.541 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:42 smithi188 ceph-mon[48761]: pgmap v668: 129 pgs: 129 active+clean; 381 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 21 MiB/s wr, 338 op/s 2024-05-08T20:21:42.706 INFO:tasks.workunit.client.0.smithi083.stdout: 10 6485 87.68 MB/sec warmup 6 sec latency 142.432 ms 2024-05-08T20:21:43.445 INFO:tasks.workunit.client.1.smithi188.stdout: 10 4011 57.23 MB/sec warmup 7 sec latency 52.532 ms 2024-05-08T20:21:43.706 INFO:tasks.workunit.client.0.smithi083.stdout: 10 7836 85.05 MB/sec warmup 7 sec latency 124.606 ms 2024-05-08T20:21:44.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:44 smithi083 ceph-mon[38695]: pgmap v669: 129 pgs: 129 active+clean; 389 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 21 MiB/s wr, 293 op/s 2024-05-08T20:21:44.445 INFO:tasks.workunit.client.1.smithi188.stdout: 10 4354 50.68 MB/sec warmup 8 sec latency 54.873 ms 2024-05-08T20:21:44.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:44 smithi188 ceph-mon[48761]: pgmap v669: 129 pgs: 129 active+clean; 389 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 21 MiB/s wr, 293 op/s 2024-05-08T20:21:44.706 INFO:tasks.workunit.client.0.smithi083.stdout: 10 7990 74.89 MB/sec warmup 8 sec latency 129.931 ms 2024-05-08T20:21:45.445 INFO:tasks.workunit.client.1.smithi188.stdout: 10 4866 47.93 MB/sec warmup 9 sec latency 48.642 ms 2024-05-08T20:21:45.706 INFO:tasks.workunit.client.0.smithi083.stdout: 10 9452 73.48 MB/sec warmup 9 sec latency 131.467 ms 2024-05-08T20:21:46.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:46 smithi083 ceph-mon[38695]: pgmap v670: 129 pgs: 129 active+clean; 389 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 23 MiB/s wr, 341 op/s 2024-05-08T20:21:46.445 INFO:tasks.workunit.client.1.smithi188.stdout: 10 5346 45.42 MB/sec warmup 10 sec latency 96.331 ms 2024-05-08T20:21:46.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:46 smithi188 ceph-mon[48761]: pgmap v670: 129 pgs: 129 active+clean; 389 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 23 MiB/s wr, 341 op/s 2024-05-08T20:21:46.707 INFO:tasks.workunit.client.0.smithi083.stdout: 10 10417 73.61 MB/sec warmup 10 sec latency 138.745 ms 2024-05-08T20:21:47.446 INFO:tasks.workunit.client.1.smithi188.stdout: 10 5869 42.50 MB/sec warmup 11 sec latency 96.676 ms 2024-05-08T20:21:47.707 INFO:tasks.workunit.client.0.smithi083.stdout: 10 11444 71.64 MB/sec warmup 11 sec latency 112.350 ms 2024-05-08T20:21:48.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:48 smithi083 ceph-mon[38695]: pgmap v671: 129 pgs: 129 active+clean; 406 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 26 MiB/s wr, 426 op/s 2024-05-08T20:21:48.445 INFO:tasks.workunit.client.1.smithi188.stdout: 10 6510 43.79 MB/sec warmup 12 sec latency 53.897 ms 2024-05-08T20:21:48.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:48 smithi188 ceph-mon[48761]: pgmap v671: 129 pgs: 129 active+clean; 406 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 26 MiB/s wr, 426 op/s 2024-05-08T20:21:48.707 INFO:tasks.workunit.client.0.smithi083.stdout: 10 12212 69.55 MB/sec warmup 12 sec latency 108.821 ms 2024-05-08T20:21:49.446 INFO:tasks.workunit.client.1.smithi188.stdout: 10 7092 44.03 MB/sec warmup 13 sec latency 50.538 ms 2024-05-08T20:21:49.707 INFO:tasks.workunit.client.0.smithi083.stdout: 10 13290 67.58 MB/sec warmup 13 sec latency 187.373 ms 2024-05-08T20:21:50.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:50 smithi083 ceph-mon[38695]: pgmap v672: 129 pgs: 129 active+clean; 396 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 26 MiB/s wr, 488 op/s 2024-05-08T20:21:50.446 INFO:tasks.workunit.client.1.smithi188.stdout: 10 7492 42.27 MB/sec warmup 14 sec latency 79.611 ms 2024-05-08T20:21:50.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:50 smithi188 ceph-mon[48761]: pgmap v672: 129 pgs: 129 active+clean; 396 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 26 MiB/s wr, 488 op/s 2024-05-08T20:21:50.707 INFO:tasks.workunit.client.0.smithi083.stdout: 10 13503 64.35 MB/sec warmup 14 sec latency 717.138 ms 2024-05-08T20:21:51.446 INFO:tasks.workunit.client.1.smithi188.stdout: 10 7871 39.73 MB/sec warmup 15 sec latency 30.041 ms 2024-05-08T20:21:51.707 INFO:tasks.workunit.client.0.smithi083.stdout: 10 13660 60.64 MB/sec warmup 15 sec latency 625.481 ms 2024-05-08T20:21:52.442 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:52 smithi083 ceph-mon[38695]: pgmap v673: 129 pgs: 129 active+clean; 398 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 16 KiB/s rd, 25 MiB/s wr, 598 op/s 2024-05-08T20:21:52.446 INFO:tasks.workunit.client.1.smithi188.stdout: 10 8285 38.28 MB/sec warmup 16 sec latency 59.311 ms 2024-05-08T20:21:52.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:52 smithi188 ceph-mon[48761]: pgmap v673: 129 pgs: 129 active+clean; 398 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 16 KiB/s rd, 25 MiB/s wr, 598 op/s 2024-05-08T20:21:52.707 INFO:tasks.workunit.client.0.smithi083.stdout: 10 14071 58.27 MB/sec warmup 16 sec latency 546.496 ms 2024-05-08T20:21:53.447 INFO:tasks.workunit.client.1.smithi188.stdout: 10 8771 37.91 MB/sec warmup 17 sec latency 74.064 ms 2024-05-08T20:21:53.709 INFO:tasks.workunit.client.0.smithi083.stdout: 10 14991 57.60 MB/sec warmup 17 sec latency 109.005 ms 2024-05-08T20:21:54.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:54 smithi083 ceph-mon[38695]: pgmap v674: 129 pgs: 129 active+clean; 400 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 16 KiB/s rd, 12 MiB/s wr, 453 op/s 2024-05-08T20:21:54.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:21:54.447 INFO:tasks.workunit.client.1.smithi188.stdout: 10 9263 36.08 MB/sec warmup 18 sec latency 86.012 ms 2024-05-08T20:21:54.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:54 smithi188 ceph-mon[48761]: pgmap v674: 129 pgs: 129 active+clean; 400 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 16 KiB/s rd, 12 MiB/s wr, 453 op/s 2024-05-08T20:21:54.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:21:54.709 INFO:tasks.workunit.client.0.smithi083.stdout: 10 15819 57.02 MB/sec warmup 18 sec latency 107.886 ms 2024-05-08T20:21:55.452 INFO:tasks.workunit.client.1.smithi188.stdout: 10 9810 36.46 MB/sec warmup 19 sec latency 46.110 ms 2024-05-08T20:21:55.709 INFO:tasks.workunit.client.0.smithi083.stdout: 10 17128 57.80 MB/sec warmup 19 sec latency 144.230 ms 2024-05-08T20:21:56.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:56 smithi083 ceph-mon[38695]: pgmap v675: 129 pgs: 129 active+clean; 404 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 16 KiB/s rd, 13 MiB/s wr, 581 op/s 2024-05-08T20:21:56.452 INFO:tasks.workunit.client.1.smithi188.stdout: 10 10431 36.79 MB/sec warmup 20 sec latency 47.003 ms 2024-05-08T20:21:56.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:56 smithi188 ceph-mon[48761]: pgmap v675: 129 pgs: 129 active+clean; 404 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 16 KiB/s rd, 13 MiB/s wr, 581 op/s 2024-05-08T20:21:56.709 INFO:tasks.workunit.client.0.smithi083.stdout: 10 18427 58.38 MB/sec warmup 20 sec latency 129.545 ms 2024-05-08T20:21:57.453 INFO:tasks.workunit.client.1.smithi188.stdout: 10 10879 36.99 MB/sec warmup 21 sec latency 56.859 ms 2024-05-08T20:21:57.709 INFO:tasks.workunit.client.0.smithi083.stdout: 10 18561 55.75 MB/sec warmup 21 sec latency 100.996 ms 2024-05-08T20:21:58.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:21:58 smithi083 ceph-mon[38695]: pgmap v676: 129 pgs: 129 active+clean; 394 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 13 MiB/s wr, 637 op/s 2024-05-08T20:21:58.452 INFO:tasks.workunit.client.1.smithi188.stdout: 10 11187 35.59 MB/sec warmup 22 sec latency 48.191 ms 2024-05-08T20:21:58.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:21:58 smithi188 ceph-mon[48761]: pgmap v676: 129 pgs: 129 active+clean; 394 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 13 MiB/s wr, 637 op/s 2024-05-08T20:21:58.709 INFO:tasks.workunit.client.0.smithi083.stdout: 10 19637 55.44 MB/sec warmup 22 sec latency 125.042 ms 2024-05-08T20:21:59.453 INFO:tasks.workunit.client.1.smithi188.stdout: 10 11645 34.57 MB/sec warmup 23 sec latency 65.615 ms 2024-05-08T20:21:59.709 INFO:tasks.workunit.client.0.smithi083.stdout: 10 21070 56.91 MB/sec warmup 23 sec latency 136.292 ms 2024-05-08T20:22:00.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:00 smithi083 ceph-mon[38695]: pgmap v677: 129 pgs: 129 active+clean; 393 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 628 op/s 2024-05-08T20:22:00.452 INFO:tasks.workunit.client.1.smithi188.stdout: 10 12163 34.75 MB/sec warmup 24 sec latency 63.383 ms 2024-05-08T20:22:00.600 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:00 smithi188 ceph-mon[48761]: pgmap v677: 129 pgs: 129 active+clean; 393 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 628 op/s 2024-05-08T20:22:00.709 INFO:tasks.workunit.client.0.smithi083.stdout: 10 22062 56.69 MB/sec warmup 24 sec latency 124.887 ms 2024-05-08T20:22:01.453 INFO:tasks.workunit.client.1.smithi188.stdout: 10 12572 33.51 MB/sec warmup 25 sec latency 133.054 ms 2024-05-08T20:22:01.710 INFO:tasks.workunit.client.0.smithi083.stdout: 10 22282 54.64 MB/sec warmup 25 sec latency 101.143 ms 2024-05-08T20:22:02.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:02 smithi083 ceph-mon[38695]: pgmap v678: 129 pgs: 129 active+clean; 417 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 23 KiB/s rd, 14 MiB/s wr, 655 op/s 2024-05-08T20:22:02.453 INFO:tasks.workunit.client.1.smithi188.stdout: 10 13065 32.87 MB/sec warmup 26 sec latency 69.569 ms 2024-05-08T20:22:02.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:02 smithi188 ceph-mon[48761]: pgmap v678: 129 pgs: 129 active+clean; 417 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 23 KiB/s rd, 14 MiB/s wr, 655 op/s 2024-05-08T20:22:02.710 INFO:tasks.workunit.client.0.smithi083.stdout: 10 23768 55.06 MB/sec warmup 26 sec latency 147.897 ms 2024-05-08T20:22:03.453 INFO:tasks.workunit.client.1.smithi188.stdout: 10 13770 33.89 MB/sec warmup 27 sec latency 68.691 ms 2024-05-08T20:22:03.710 INFO:tasks.workunit.client.0.smithi083.stdout: 10 24745 55.79 MB/sec warmup 27 sec latency 319.808 ms 2024-05-08T20:22:04.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:04 smithi083 ceph-mon[38695]: pgmap v679: 129 pgs: 129 active+clean; 415 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 497 op/s 2024-05-08T20:22:04.453 INFO:tasks.workunit.client.1.smithi188.stdout: 10 14240 34.08 MB/sec warmup 28 sec latency 48.035 ms 2024-05-08T20:22:04.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:04 smithi188 ceph-mon[48761]: pgmap v679: 129 pgs: 129 active+clean; 415 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 497 op/s 2024-05-08T20:22:04.710 INFO:tasks.workunit.client.0.smithi083.stdout: 10 25646 55.45 MB/sec warmup 28 sec latency 111.086 ms 2024-05-08T20:22:05.453 INFO:tasks.workunit.client.1.smithi188.stdout: 10 14702 33.59 MB/sec warmup 29 sec latency 63.937 ms 2024-05-08T20:22:05.710 INFO:tasks.workunit.client.0.smithi083.stdout: 10 26114 54.43 MB/sec warmup 29 sec latency 128.124 ms 2024-05-08T20:22:06.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:06 smithi083 ceph-mon[38695]: pgmap v680: 129 pgs: 129 active+clean; 409 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 13 MiB/s wr, 594 op/s 2024-05-08T20:22:06.453 INFO:tasks.workunit.client.1.smithi188.stdout: 10 15200 32.93 MB/sec warmup 30 sec latency 77.289 ms 2024-05-08T20:22:06.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:06 smithi188 ceph-mon[48761]: pgmap v680: 129 pgs: 129 active+clean; 409 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 13 MiB/s wr, 594 op/s 2024-05-08T20:22:06.710 INFO:tasks.workunit.client.0.smithi083.stdout: 10 27351 54.21 MB/sec warmup 30 sec latency 167.982 ms 2024-05-08T20:22:07.458 INFO:tasks.workunit.client.1.smithi188.stdout: 10 15783 33.09 MB/sec warmup 31 sec latency 67.103 ms 2024-05-08T20:22:07.710 INFO:tasks.workunit.client.0.smithi083.stdout: 10 29096 56.18 MB/sec warmup 31 sec latency 121.578 ms 2024-05-08T20:22:08.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:08 smithi188 ceph-mon[48761]: pgmap v681: 129 pgs: 129 active+clean; 406 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 521 op/s 2024-05-08T20:22:08.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:08 smithi083 ceph-mon[38695]: pgmap v681: 129 pgs: 129 active+clean; 406 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 521 op/s 2024-05-08T20:22:08.453 INFO:tasks.workunit.client.1.smithi188.stdout: 10 16250 32.20 MB/sec warmup 32 sec latency 92.578 ms 2024-05-08T20:22:08.710 INFO:tasks.workunit.client.0.smithi083.stdout: 10 29165 54.48 MB/sec warmup 32 sec latency 262.691 ms 2024-05-08T20:22:09.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:22:09.453 INFO:tasks.workunit.client.1.smithi188.stdout: 10 16729 31.87 MB/sec warmup 33 sec latency 91.207 ms 2024-05-08T20:22:09.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:22:09.710 INFO:tasks.workunit.client.0.smithi083.stdout: 10 29218 52.87 MB/sec warmup 33 sec latency 462.826 ms 2024-05-08T20:22:10.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:10 smithi083 ceph-mon[38695]: pgmap v682: 129 pgs: 129 active+clean; 408 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 504 op/s 2024-05-08T20:22:10.453 INFO:tasks.workunit.client.1.smithi188.stdout: 10 17508 32.89 MB/sec warmup 34 sec latency 55.688 ms 2024-05-08T20:22:10.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:10 smithi188 ceph-mon[48761]: pgmap v682: 129 pgs: 129 active+clean; 408 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 504 op/s 2024-05-08T20:22:10.710 INFO:tasks.workunit.client.0.smithi083.stdout: 10 29223 51.32 MB/sec warmup 34 sec latency 551.314 ms 2024-05-08T20:22:11.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:11 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:22:11.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:11 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:22:11.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:11 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:22:11.453 INFO:tasks.workunit.client.1.smithi188.stdout: 10 17981 33.12 MB/sec warmup 35 sec latency 46.612 ms 2024-05-08T20:22:11.599 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:11 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:22:11.599 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:11 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:22:11.599 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:11 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:22:11.710 INFO:tasks.workunit.client.0.smithi083.stdout: 10 30283 51.25 MB/sec warmup 35 sec latency 529.833 ms 2024-05-08T20:22:12.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:12 smithi083 ceph-mon[38695]: pgmap v683: 129 pgs: 129 active+clean; 412 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 23 KiB/s rd, 14 MiB/s wr, 596 op/s 2024-05-08T20:22:12.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:12 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:22:12.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:12 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:22:12.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:12 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:22:12.453 INFO:tasks.workunit.client.1.smithi188.stdout: 10 18380 32.40 MB/sec warmup 36 sec latency 61.798 ms 2024-05-08T20:22:12.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:12 smithi188 ceph-mon[48761]: pgmap v683: 129 pgs: 129 active+clean; 412 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 23 KiB/s rd, 14 MiB/s wr, 596 op/s 2024-05-08T20:22:12.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:12 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:22:12.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:12 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:22:12.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:12 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:22:12.710 INFO:tasks.workunit.client.0.smithi083.stdout: 10 31523 51.95 MB/sec warmup 36 sec latency 132.262 ms 2024-05-08T20:22:13.454 INFO:tasks.workunit.client.1.smithi188.stdout: 10 18854 31.98 MB/sec warmup 37 sec latency 49.585 ms 2024-05-08T20:22:13.710 INFO:tasks.workunit.client.0.smithi083.stdout: 10 32698 52.28 MB/sec warmup 37 sec latency 131.340 ms 2024-05-08T20:22:14.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:14 smithi083 ceph-mon[38695]: pgmap v684: 129 pgs: 129 active+clean; 413 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 497 op/s 2024-05-08T20:22:14.455 INFO:tasks.workunit.client.1.smithi188.stdout: 10 19388 32.04 MB/sec warmup 38 sec latency 59.414 ms 2024-05-08T20:22:14.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:14 smithi188 ceph-mon[48761]: pgmap v684: 129 pgs: 129 active+clean; 413 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 497 op/s 2024-05-08T20:22:14.710 INFO:tasks.workunit.client.0.smithi083.stdout: 10 32869 51.01 MB/sec warmup 38 sec latency 117.266 ms 2024-05-08T20:22:15.454 INFO:tasks.workunit.client.1.smithi188.stdout: 10 19832 31.34 MB/sec warmup 39 sec latency 82.159 ms 2024-05-08T20:22:15.710 INFO:tasks.workunit.client.0.smithi083.stdout: 10 34330 51.30 MB/sec warmup 39 sec latency 132.928 ms 2024-05-08T20:22:16.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:16 smithi083 ceph-mon[38695]: pgmap v685: 129 pgs: 129 active+clean; 412 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 17 KiB/s rd, 13 MiB/s wr, 540 op/s 2024-05-08T20:22:16.454 INFO:tasks.workunit.client.1.smithi188.stdout: 10 20396 31.35 MB/sec warmup 40 sec latency 71.298 ms 2024-05-08T20:22:16.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:16 smithi188 ceph-mon[48761]: pgmap v685: 129 pgs: 129 active+clean; 412 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 17 KiB/s rd, 13 MiB/s wr, 540 op/s 2024-05-08T20:22:16.711 INFO:tasks.workunit.client.0.smithi083.stdout: 10 35387 52.00 MB/sec warmup 40 sec latency 131.086 ms 2024-05-08T20:22:17.454 INFO:tasks.workunit.client.1.smithi188.stdout: 10 21120 31.96 MB/sec warmup 41 sec latency 39.203 ms 2024-05-08T20:22:17.710 INFO:tasks.workunit.client.0.smithi083.stdout: 10 36279 51.86 MB/sec warmup 41 sec latency 97.659 ms 2024-05-08T20:22:18.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:18 smithi083 ceph-mon[38695]: pgmap v686: 129 pgs: 129 active+clean; 414 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 26 KiB/s rd, 13 MiB/s wr, 594 op/s 2024-05-08T20:22:18.454 INFO:tasks.workunit.client.1.smithi188.stdout: 10 21647 32.26 MB/sec warmup 42 sec latency 43.618 ms 2024-05-08T20:22:18.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:18 smithi188 ceph-mon[48761]: pgmap v686: 129 pgs: 129 active+clean; 414 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 26 KiB/s rd, 13 MiB/s wr, 594 op/s 2024-05-08T20:22:18.710 INFO:tasks.workunit.client.0.smithi083.stdout: 10 37021 51.65 MB/sec warmup 42 sec latency 111.005 ms 2024-05-08T20:22:19.454 INFO:tasks.workunit.client.1.smithi188.stdout: 10 22061 31.63 MB/sec warmup 43 sec latency 61.500 ms 2024-05-08T20:22:19.710 INFO:tasks.workunit.client.0.smithi083.stdout: 10 38320 52.01 MB/sec warmup 43 sec latency 123.324 ms 2024-05-08T20:22:20.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:20 smithi083 ceph-mon[38695]: pgmap v687: 129 pgs: 129 active+clean; 411 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 615 op/s 2024-05-08T20:22:20.454 INFO:tasks.workunit.client.1.smithi188.stdout: 10 22495 31.37 MB/sec warmup 44 sec latency 81.640 ms 2024-05-08T20:22:20.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:20 smithi188 ceph-mon[48761]: pgmap v687: 129 pgs: 129 active+clean; 411 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 615 op/s 2024-05-08T20:22:20.710 INFO:tasks.workunit.client.0.smithi083.stdout: 10 39739 52.62 MB/sec warmup 44 sec latency 125.015 ms 2024-05-08T20:22:21.454 INFO:tasks.workunit.client.1.smithi188.stdout: 10 23076 31.34 MB/sec warmup 45 sec latency 25.726 ms 2024-05-08T20:22:21.710 INFO:tasks.workunit.client.0.smithi083.stdout: 10 39866 51.52 MB/sec warmup 45 sec latency 108.584 ms 2024-05-08T20:22:22.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:22 smithi083 ceph-mon[38695]: pgmap v688: 129 pgs: 129 active+clean; 421 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 26 KiB/s rd, 15 MiB/s wr, 715 op/s 2024-05-08T20:22:22.454 INFO:tasks.workunit.client.1.smithi188.stdout: 10 23524 30.82 MB/sec warmup 46 sec latency 106.467 ms 2024-05-08T20:22:22.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:22 smithi188 ceph-mon[48761]: pgmap v688: 129 pgs: 129 active+clean; 421 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 26 KiB/s rd, 15 MiB/s wr, 715 op/s 2024-05-08T20:22:22.710 INFO:tasks.workunit.client.0.smithi083.stdout: 10 40948 51.46 MB/sec warmup 46 sec latency 125.590 ms 2024-05-08T20:22:23.454 INFO:tasks.workunit.client.1.smithi188.stdout: 10 24073 31.04 MB/sec warmup 47 sec latency 69.211 ms 2024-05-08T20:22:23.712 INFO:tasks.workunit.client.0.smithi083.stdout: 10 42280 52.15 MB/sec warmup 47 sec latency 137.909 ms 2024-05-08T20:22:24.454 INFO:tasks.workunit.client.1.smithi188.stdout: 10 24726 31.42 MB/sec warmup 48 sec latency 64.065 ms 2024-05-08T20:22:24.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:24 smithi188 ceph-mon[48761]: pgmap v689: 129 pgs: 129 active+clean; 418 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 507 op/s 2024-05-08T20:22:24.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:24 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:22:24.694 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:24 smithi083 ceph-mon[38695]: pgmap v689: 129 pgs: 129 active+clean; 418 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 507 op/s 2024-05-08T20:22:24.694 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:22:24.712 INFO:tasks.workunit.client.0.smithi083.stdout: 10 43333 52.24 MB/sec warmup 48 sec latency 117.715 ms 2024-05-08T20:22:25.454 INFO:tasks.workunit.client.1.smithi188.stdout: 10 25244 31.58 MB/sec warmup 49 sec latency 45.854 ms 2024-05-08T20:22:25.712 INFO:tasks.workunit.client.0.smithi083.stdout: 10 43518 51.27 MB/sec warmup 49 sec latency 103.365 ms 2024-05-08T20:22:26.454 INFO:tasks.workunit.client.1.smithi188.stdout: 10 25605 31.02 MB/sec warmup 50 sec latency 64.544 ms 2024-05-08T20:22:26.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:26 smithi188 ceph-mon[48761]: pgmap v690: 129 pgs: 129 active+clean; 408 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 16 KiB/s rd, 13 MiB/s wr, 595 op/s 2024-05-08T20:22:26.694 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:26 smithi083 ceph-mon[38695]: pgmap v690: 129 pgs: 129 active+clean; 408 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 16 KiB/s rd, 13 MiB/s wr, 595 op/s 2024-05-08T20:22:26.712 INFO:tasks.workunit.client.0.smithi083.stdout: 10 43852 50.69 MB/sec warmup 50 sec latency 576.490 ms 2024-05-08T20:22:27.454 INFO:tasks.workunit.client.1.smithi188.stdout: 10 26037 30.76 MB/sec warmup 51 sec latency 67.967 ms 2024-05-08T20:22:27.712 INFO:tasks.workunit.client.0.smithi083.stdout: 10 44261 50.13 MB/sec warmup 51 sec latency 601.989 ms 2024-05-08T20:22:28.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:28 smithi083 ceph-mon[38695]: pgmap v691: 129 pgs: 129 active+clean; 413 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 23 KiB/s rd, 14 MiB/s wr, 672 op/s 2024-05-08T20:22:28.454 INFO:tasks.workunit.client.1.smithi188.stdout: 10 26529 30.78 MB/sec warmup 52 sec latency 71.864 ms 2024-05-08T20:22:28.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:28 smithi188 ceph-mon[48761]: pgmap v691: 129 pgs: 129 active+clean; 413 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 23 KiB/s rd, 14 MiB/s wr, 672 op/s 2024-05-08T20:22:28.712 INFO:tasks.workunit.client.0.smithi083.stdout: 10 44717 49.35 MB/sec warmup 52 sec latency 434.130 ms 2024-05-08T20:22:29.456 INFO:tasks.workunit.client.1.smithi188.stdout: 10 27027 30.31 MB/sec warmup 53 sec latency 86.020 ms 2024-05-08T20:22:29.712 INFO:tasks.workunit.client.0.smithi083.stdout: 10 45041 48.65 MB/sec warmup 53 sec latency 320.018 ms 2024-05-08T20:22:30.454 INFO:tasks.workunit.client.1.smithi188.stdout: 10 27629 30.71 MB/sec warmup 54 sec latency 69.347 ms 2024-05-08T20:22:30.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:30 smithi188 ceph-mon[48761]: pgmap v692: 129 pgs: 129 active+clean; 417 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 14 KiB/s rd, 13 MiB/s wr, 547 op/s 2024-05-08T20:22:30.694 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:30 smithi083 ceph-mon[38695]: pgmap v692: 129 pgs: 129 active+clean; 417 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 14 KiB/s rd, 13 MiB/s wr, 547 op/s 2024-05-08T20:22:30.712 INFO:tasks.workunit.client.0.smithi083.stdout: 10 45531 48.67 MB/sec warmup 54 sec latency 318.615 ms 2024-05-08T20:22:31.455 INFO:tasks.workunit.client.1.smithi188.stdout: 10 28257 30.89 MB/sec warmup 55 sec latency 61.871 ms 2024-05-08T20:22:31.712 INFO:tasks.workunit.client.0.smithi083.stdout: 10 45886 48.13 MB/sec warmup 55 sec latency 391.898 ms 2024-05-08T20:22:32.455 INFO:tasks.workunit.client.1.smithi188.stdout: 10 28744 31.03 MB/sec warmup 56 sec latency 55.857 ms 2024-05-08T20:22:32.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:32 smithi188 ceph-mon[48761]: pgmap v693: 129 pgs: 129 active+clean; 401 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 23 KiB/s rd, 14 MiB/s wr, 675 op/s 2024-05-08T20:22:32.694 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:32 smithi083 ceph-mon[38695]: pgmap v693: 129 pgs: 129 active+clean; 401 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 23 KiB/s rd, 14 MiB/s wr, 675 op/s 2024-05-08T20:22:32.712 INFO:tasks.workunit.client.0.smithi083.stdout: 10 46886 48.20 MB/sec warmup 56 sec latency 119.976 ms 2024-05-08T20:22:33.455 INFO:tasks.workunit.client.1.smithi188.stdout: 10 29146 30.57 MB/sec warmup 57 sec latency 88.102 ms 2024-05-08T20:22:33.713 INFO:tasks.workunit.client.0.smithi083.stdout: 10 47132 47.45 MB/sec warmup 57 sec latency 105.490 ms 2024-05-08T20:22:34.455 INFO:tasks.workunit.client.1.smithi188.stdout: 10 29623 30.40 MB/sec warmup 58 sec latency 67.678 ms 2024-05-08T20:22:34.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:34 smithi188 ceph-mon[48761]: pgmap v694: 129 pgs: 129 active+clean; 401 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 16 KiB/s rd, 9.8 MiB/s wr, 494 op/s 2024-05-08T20:22:34.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:34 smithi083 ceph-mon[38695]: pgmap v694: 129 pgs: 129 active+clean; 401 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 16 KiB/s rd, 9.8 MiB/s wr, 494 op/s 2024-05-08T20:22:34.713 INFO:tasks.workunit.client.0.smithi083.stdout: 10 48762 48.08 MB/sec warmup 58 sec latency 143.141 ms 2024-05-08T20:22:35.455 INFO:tasks.workunit.client.1.smithi188.stdout: 10 30137 30.38 MB/sec warmup 59 sec latency 104.280 ms 2024-05-08T20:22:35.713 INFO:tasks.workunit.client.0.smithi083.stdout: 10 50369 48.96 MB/sec warmup 59 sec latency 114.156 ms 2024-05-08T20:22:36.455 INFO:tasks.workunit.client.1.smithi188.stdout: 10 30592 29.98 MB/sec warmup 60 sec latency 82.003 ms 2024-05-08T20:22:36.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:36 smithi188 ceph-mon[48761]: pgmap v695: 129 pgs: 129 active+clean; 410 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 16 KiB/s rd, 12 MiB/s wr, 589 op/s 2024-05-08T20:22:36.694 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:36 smithi083 ceph-mon[38695]: pgmap v695: 129 pgs: 129 active+clean; 410 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 16 KiB/s rd, 12 MiB/s wr, 589 op/s 2024-05-08T20:22:36.713 INFO:tasks.workunit.client.0.smithi083.stdout: 10 50493 48.20 MB/sec warmup 60 sec latency 122.907 ms 2024-05-08T20:22:37.455 INFO:tasks.workunit.client.1.smithi188.stdout: 10 31147 30.18 MB/sec warmup 61 sec latency 82.357 ms 2024-05-08T20:22:37.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:37 smithi188 ceph-mon[48761]: pgmap v696: 129 pgs: 129 active+clean; 423 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 22 KiB/s rd, 14 MiB/s wr, 630 op/s 2024-05-08T20:22:37.694 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:37 smithi083 ceph-mon[38695]: pgmap v696: 129 pgs: 129 active+clean; 423 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 22 KiB/s rd, 14 MiB/s wr, 630 op/s 2024-05-08T20:22:37.713 INFO:tasks.workunit.client.0.smithi083.stdout: 10 50989 47.83 MB/sec warmup 61 sec latency 325.453 ms 2024-05-08T20:22:38.455 INFO:tasks.workunit.client.1.smithi188.stdout: 10 31812 30.49 MB/sec warmup 62 sec latency 53.539 ms 2024-05-08T20:22:38.713 INFO:tasks.workunit.client.0.smithi083.stdout: 10 52503 48.40 MB/sec warmup 62 sec latency 153.629 ms 2024-05-08T20:22:39.455 INFO:tasks.workunit.client.1.smithi188.stdout: 10 32281 30.60 MB/sec warmup 63 sec latency 73.556 ms 2024-05-08T20:22:39.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:39 smithi188 ceph-mon[48761]: pgmap v697: 129 pgs: 129 active+clean; 427 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 16 KiB/s rd, 12 MiB/s wr, 585 op/s 2024-05-08T20:22:39.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:22:39.694 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:39 smithi083 ceph-mon[38695]: pgmap v697: 129 pgs: 129 active+clean; 427 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 16 KiB/s rd, 12 MiB/s wr, 585 op/s 2024-05-08T20:22:39.694 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:22:39.713 INFO:tasks.workunit.client.0.smithi083.stdout: 10 53918 48.89 MB/sec warmup 63 sec latency 117.017 ms 2024-05-08T20:22:40.455 INFO:tasks.workunit.client.1.smithi188.stdout: 10 32679 30.21 MB/sec warmup 64 sec latency 60.085 ms 2024-05-08T20:22:40.713 INFO:tasks.workunit.client.0.smithi083.stdout: 10 54044 48.17 MB/sec warmup 64 sec latency 129.021 ms 2024-05-08T20:22:41.456 INFO:tasks.workunit.client.1.smithi188.stdout: 10 33167 30.05 MB/sec warmup 65 sec latency 26.256 ms 2024-05-08T20:22:41.594 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:41 smithi188 ceph-mon[48761]: pgmap v698: 129 pgs: 129 active+clean; 423 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 24 KiB/s rd, 15 MiB/s wr, 661 op/s 2024-05-08T20:22:41.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:41 smithi083 ceph-mon[38695]: pgmap v698: 129 pgs: 129 active+clean; 423 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 24 KiB/s rd, 15 MiB/s wr, 661 op/s 2024-05-08T20:22:41.713 INFO:tasks.workunit.client.0.smithi083.stdout: 10 55102 48.18 MB/sec warmup 65 sec latency 113.878 ms 2024-05-08T20:22:42.456 INFO:tasks.workunit.client.1.smithi188.stdout: 10 33628 30.05 MB/sec warmup 66 sec latency 93.196 ms 2024-05-08T20:22:42.713 INFO:tasks.workunit.client.0.smithi083.stdout: 10 56051 48.37 MB/sec warmup 66 sec latency 369.342 ms 2024-05-08T20:22:43.456 INFO:tasks.workunit.client.1.smithi188.stdout: 10 34056 29.68 MB/sec warmup 67 sec latency 118.794 ms 2024-05-08T20:22:43.598 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:43 smithi188 ceph-mon[48761]: pgmap v699: 129 pgs: 129 active+clean; 426 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 15 KiB/s rd, 12 MiB/s wr, 497 op/s 2024-05-08T20:22:43.694 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:43 smithi083 ceph-mon[38695]: pgmap v699: 129 pgs: 129 active+clean; 426 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 15 KiB/s rd, 12 MiB/s wr, 497 op/s 2024-05-08T20:22:43.713 INFO:tasks.workunit.client.0.smithi083.stdout: 10 57460 48.82 MB/sec warmup 67 sec latency 374.540 ms 2024-05-08T20:22:44.456 INFO:tasks.workunit.client.1.smithi188.stdout: 10 34512 29.58 MB/sec warmup 68 sec latency 86.854 ms 2024-05-08T20:22:44.713 INFO:tasks.workunit.client.0.smithi083.stdout: 10 57584 48.15 MB/sec warmup 68 sec latency 115.568 ms 2024-05-08T20:22:45.456 INFO:tasks.workunit.client.1.smithi188.stdout: 10 35262 30.05 MB/sec warmup 69 sec latency 65.425 ms 2024-05-08T20:22:45.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:45 smithi188 ceph-mon[48761]: pgmap v700: 129 pgs: 129 active+clean; 420 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 15 KiB/s rd, 14 MiB/s wr, 625 op/s 2024-05-08T20:22:45.694 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:45 smithi083 ceph-mon[38695]: pgmap v700: 129 pgs: 129 active+clean; 420 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 15 KiB/s rd, 14 MiB/s wr, 625 op/s 2024-05-08T20:22:45.713 INFO:tasks.workunit.client.0.smithi083.stdout: 10 58404 48.13 MB/sec warmup 69 sec latency 127.848 ms 2024-05-08T20:22:46.456 INFO:tasks.workunit.client.1.smithi188.stdout: 10 35733 30.23 MB/sec warmup 70 sec latency 37.458 ms 2024-05-08T20:22:46.713 INFO:tasks.workunit.client.0.smithi083.stdout: 10 59761 48.50 MB/sec warmup 70 sec latency 127.461 ms 2024-05-08T20:22:47.456 INFO:tasks.workunit.client.1.smithi188.stdout: 10 36118 29.90 MB/sec warmup 71 sec latency 62.020 ms 2024-05-08T20:22:47.599 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:47 smithi188 ceph-mon[48761]: pgmap v701: 129 pgs: 129 active+clean; 414 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 23 KiB/s rd, 13 MiB/s wr, 587 op/s 2024-05-08T20:22:47.694 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:47 smithi083 ceph-mon[38695]: pgmap v701: 129 pgs: 129 active+clean; 414 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 23 KiB/s rd, 13 MiB/s wr, 587 op/s 2024-05-08T20:22:47.713 INFO:tasks.workunit.client.0.smithi083.stdout: 10 61039 48.78 MB/sec warmup 71 sec latency 137.400 ms 2024-05-08T20:22:48.456 INFO:tasks.workunit.client.1.smithi188.stdout: 10 36617 29.73 MB/sec warmup 72 sec latency 78.340 ms 2024-05-08T20:22:48.714 INFO:tasks.workunit.client.0.smithi083.stdout: 10 61137 48.13 MB/sec warmup 72 sec latency 246.047 ms 2024-05-08T20:22:49.457 INFO:tasks.workunit.client.1.smithi188.stdout: 10 37148 29.78 MB/sec warmup 73 sec latency 106.500 ms 2024-05-08T20:22:49.694 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:49 smithi083 ceph-mon[38695]: pgmap v702: 129 pgs: 129 active+clean; 416 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 555 op/s 2024-05-08T20:22:49.716 INFO:tasks.workunit.client.0.smithi083.stdout: 10 62068 48.13 MB/sec warmup 73 sec latency 120.837 ms 2024-05-08T20:22:49.849 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:49 smithi188 ceph-mon[48761]: pgmap v702: 129 pgs: 129 active+clean; 416 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 555 op/s 2024-05-08T20:22:50.456 INFO:tasks.workunit.client.1.smithi188.stdout: 10 37589 29.44 MB/sec warmup 74 sec latency 93.221 ms 2024-05-08T20:22:50.715 INFO:tasks.workunit.client.0.smithi083.stdout: 10 63356 48.48 MB/sec warmup 74 sec latency 133.291 ms 2024-05-08T20:22:51.456 INFO:tasks.workunit.client.1.smithi188.stdout: 10 38098 29.40 MB/sec warmup 75 sec latency 82.304 ms 2024-05-08T20:22:51.694 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:51 smithi083 ceph-mon[38695]: pgmap v703: 129 pgs: 129 active+clean; 424 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 26 KiB/s rd, 14 MiB/s wr, 626 op/s 2024-05-08T20:22:51.715 INFO:tasks.workunit.client.0.smithi083.stdout: 10 64580 48.72 MB/sec warmup 75 sec latency 120.915 ms 2024-05-08T20:22:51.845 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:51 smithi188 ceph-mon[48761]: pgmap v703: 129 pgs: 129 active+clean; 424 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 26 KiB/s rd, 14 MiB/s wr, 626 op/s 2024-05-08T20:22:52.456 INFO:tasks.workunit.client.1.smithi188.stdout: 10 38831 29.82 MB/sec warmup 76 sec latency 48.626 ms 2024-05-08T20:22:52.715 INFO:tasks.workunit.client.0.smithi083.stdout: 10 64747 48.13 MB/sec warmup 76 sec latency 108.853 ms 2024-05-08T20:22:53.456 INFO:tasks.workunit.client.1.smithi188.stdout: 10 39307 29.99 MB/sec warmup 77 sec latency 40.949 ms 2024-05-08T20:22:53.694 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:53 smithi083 ceph-mon[38695]: pgmap v704: 129 pgs: 129 active+clean; 423 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 533 op/s 2024-05-08T20:22:53.713 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:53 smithi188 ceph-mon[48761]: pgmap v704: 129 pgs: 129 active+clean; 423 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 533 op/s 2024-05-08T20:22:53.715 INFO:tasks.workunit.client.0.smithi083.stdout: 10 65945 48.20 MB/sec warmup 77 sec latency 122.261 ms 2024-05-08T20:22:54.457 INFO:tasks.workunit.client.1.smithi188.stdout: 10 39641 29.66 MB/sec warmup 78 sec latency 77.143 ms 2024-05-08T20:22:54.694 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:22:54.715 INFO:tasks.workunit.client.0.smithi083.stdout: 10 67173 48.66 MB/sec warmup 78 sec latency 126.217 ms 2024-05-08T20:22:54.845 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:22:55.457 INFO:tasks.workunit.client.1.smithi188.stdout: 10 40037 29.41 MB/sec warmup 79 sec latency 45.515 ms 2024-05-08T20:22:55.694 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:55 smithi083 ceph-mon[38695]: pgmap v705: 129 pgs: 129 active+clean; 424 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 17 KiB/s rd, 12 MiB/s wr, 591 op/s 2024-05-08T20:22:55.716 INFO:tasks.workunit.client.0.smithi083.stdout: 10 68159 48.69 MB/sec warmup 79 sec latency 119.526 ms 2024-05-08T20:22:55.845 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:55 smithi188 ceph-mon[48761]: pgmap v705: 129 pgs: 129 active+clean; 424 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 17 KiB/s rd, 12 MiB/s wr, 591 op/s 2024-05-08T20:22:56.459 INFO:tasks.workunit.client.1.smithi188.stdout: 10 40476 29.40 MB/sec warmup 80 sec latency 110.074 ms 2024-05-08T20:22:56.716 INFO:tasks.workunit.client.0.smithi083.stdout: 10 68371 48.14 MB/sec warmup 80 sec latency 116.627 ms 2024-05-08T20:22:57.459 INFO:tasks.workunit.client.1.smithi188.stdout: 10 40835 29.22 MB/sec warmup 81 sec latency 118.351 ms 2024-05-08T20:22:57.694 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:57 smithi083 ceph-mon[38695]: pgmap v706: 129 pgs: 129 active+clean; 426 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 22 KiB/s rd, 13 MiB/s wr, 545 op/s 2024-05-08T20:22:57.716 INFO:tasks.workunit.client.0.smithi083.stdout: 10 69809 48.31 MB/sec warmup 81 sec latency 129.398 ms 2024-05-08T20:22:57.844 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:57 smithi188 ceph-mon[48761]: pgmap v706: 129 pgs: 129 active+clean; 426 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 22 KiB/s rd, 13 MiB/s wr, 545 op/s 2024-05-08T20:22:58.459 INFO:tasks.workunit.client.1.smithi188.stdout: 10 41370 29.01 MB/sec warmup 82 sec latency 70.509 ms 2024-05-08T20:22:58.716 INFO:tasks.workunit.client.0.smithi083.stdout: 10 71107 48.86 MB/sec warmup 82 sec latency 146.210 ms 2024-05-08T20:22:59.459 INFO:tasks.workunit.client.1.smithi188.stdout: 10 42020 29.34 MB/sec warmup 83 sec latency 55.888 ms 2024-05-08T20:22:59.694 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:22:59 smithi083 ceph-mon[38695]: pgmap v707: 129 pgs: 129 active+clean; 415 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 14 KiB/s rd, 12 MiB/s wr, 560 op/s 2024-05-08T20:22:59.716 INFO:tasks.workunit.client.0.smithi083.stdout: 10 71705 48.64 MB/sec warmup 83 sec latency 127.691 ms 2024-05-08T20:22:59.845 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:22:59 smithi188 ceph-mon[48761]: pgmap v707: 129 pgs: 129 active+clean; 415 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 14 KiB/s rd, 12 MiB/s wr, 560 op/s 2024-05-08T20:23:00.459 INFO:tasks.workunit.client.1.smithi188.stdout: 10 42587 29.50 MB/sec warmup 84 sec latency 39.660 ms 2024-05-08T20:23:00.716 INFO:tasks.workunit.client.0.smithi083.stdout: 10 71946 48.13 MB/sec warmup 84 sec latency 117.473 ms 2024-05-08T20:23:01.459 INFO:tasks.workunit.client.1.smithi188.stdout: 10 42980 29.45 MB/sec warmup 85 sec latency 71.327 ms 2024-05-08T20:23:01.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:01 smithi083 ceph-mon[38695]: pgmap v708: 129 pgs: 129 active+clean; 412 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 22 KiB/s rd, 14 MiB/s wr, 622 op/s 2024-05-08T20:23:01.716 INFO:tasks.workunit.client.0.smithi083.stdout: 10 73406 48.31 MB/sec warmup 85 sec latency 141.300 ms 2024-05-08T20:23:01.845 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:01 smithi188 ceph-mon[48761]: pgmap v708: 129 pgs: 129 active+clean; 412 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 22 KiB/s rd, 14 MiB/s wr, 622 op/s 2024-05-08T20:23:02.459 INFO:tasks.workunit.client.1.smithi188.stdout: 10 43308 29.15 MB/sec warmup 86 sec latency 65.755 ms 2024-05-08T20:23:02.717 INFO:tasks.workunit.client.0.smithi083.stdout: 10 75132 49.10 MB/sec warmup 86 sec latency 148.933 ms 2024-05-08T20:23:03.459 INFO:tasks.workunit.client.1.smithi188.stdout: 10 43579 28.90 MB/sec warmup 87 sec latency 123.668 ms 2024-05-08T20:23:03.694 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:03 smithi083 ceph-mon[38695]: pgmap v709: 129 pgs: 129 active+clean; 414 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 475 op/s 2024-05-08T20:23:03.718 INFO:tasks.workunit.client.0.smithi083.stdout: 10 75316 48.62 MB/sec warmup 87 sec latency 124.880 ms 2024-05-08T20:23:03.847 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:03 smithi188 ceph-mon[48761]: pgmap v709: 129 pgs: 129 active+clean; 414 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 475 op/s 2024-05-08T20:23:04.459 INFO:tasks.workunit.client.1.smithi188.stdout: 10 44047 28.97 MB/sec warmup 88 sec latency 54.515 ms 2024-05-08T20:23:04.718 INFO:tasks.workunit.client.0.smithi083.stdout: 10 75464 48.11 MB/sec warmup 88 sec latency 385.139 ms 2024-05-08T20:23:05.460 INFO:tasks.workunit.client.1.smithi188.stdout: 10 44437 28.74 MB/sec warmup 89 sec latency 89.977 ms 2024-05-08T20:23:05.694 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:05 smithi083 ceph-mon[38695]: pgmap v710: 129 pgs: 129 active+clean; 433 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 13 MiB/s wr, 576 op/s 2024-05-08T20:23:05.718 INFO:tasks.workunit.client.0.smithi083.stdout: 10 75502 47.58 MB/sec warmup 89 sec latency 687.803 ms 2024-05-08T20:23:05.845 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:05 smithi188 ceph-mon[48761]: pgmap v710: 129 pgs: 129 active+clean; 433 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 13 MiB/s wr, 576 op/s 2024-05-08T20:23:06.460 INFO:tasks.workunit.client.1.smithi188.stdout: 10 44946 28.59 MB/sec warmup 90 sec latency 65.787 ms 2024-05-08T20:23:06.718 INFO:tasks.workunit.client.0.smithi083.stdout: 10 75783 47.29 MB/sec warmup 90 sec latency 717.768 ms 2024-05-08T20:23:07.460 INFO:tasks.workunit.client.1.smithi188.stdout: 10 45558 28.87 MB/sec warmup 91 sec latency 58.815 ms 2024-05-08T20:23:07.694 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:07 smithi083 ceph-mon[38695]: pgmap v711: 129 pgs: 129 active+clean; 422 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 22 KiB/s rd, 12 MiB/s wr, 521 op/s 2024-05-08T20:23:07.718 INFO:tasks.workunit.client.0.smithi083.stdout: 10 76167 47.01 MB/sec warmup 91 sec latency 506.359 ms 2024-05-08T20:23:07.845 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:07 smithi188 ceph-mon[48761]: pgmap v711: 129 pgs: 129 active+clean; 422 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 22 KiB/s rd, 12 MiB/s wr, 521 op/s 2024-05-08T20:23:08.460 INFO:tasks.workunit.client.1.smithi188.stdout: 10 46067 28.94 MB/sec warmup 92 sec latency 61.995 ms 2024-05-08T20:23:08.718 INFO:tasks.workunit.client.0.smithi083.stdout: 10 76409 46.52 MB/sec warmup 92 sec latency 529.037 ms 2024-05-08T20:23:09.194 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:23:09.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:23:09.460 INFO:tasks.workunit.client.1.smithi188.stdout: 10 46522 28.97 MB/sec warmup 93 sec latency 70.907 ms 2024-05-08T20:23:09.720 INFO:tasks.workunit.client.0.smithi083.stdout: 10 77399 46.70 MB/sec warmup 93 sec latency 410.566 ms 2024-05-08T20:23:10.194 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:09 smithi083 ceph-mon[38695]: pgmap v712: 129 pgs: 129 active+clean; 422 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 566 op/s 2024-05-08T20:23:10.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:09 smithi188 ceph-mon[48761]: pgmap v712: 129 pgs: 129 active+clean; 422 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 566 op/s 2024-05-08T20:23:10.460 INFO:tasks.workunit.client.1.smithi188.stdout: 10 46919 28.73 MB/sec warmup 94 sec latency 79.159 ms 2024-05-08T20:23:10.720 INFO:tasks.workunit.client.0.smithi083.stdout: 10 78773 47.01 MB/sec warmup 94 sec latency 126.847 ms 2024-05-08T20:23:11.460 INFO:tasks.workunit.client.1.smithi188.stdout: 10 47434 28.69 MB/sec warmup 95 sec latency 28.233 ms 2024-05-08T20:23:11.720 INFO:tasks.workunit.client.0.smithi083.stdout: 10 78935 46.56 MB/sec warmup 95 sec latency 125.302 ms 2024-05-08T20:23:12.194 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:11 smithi083 ceph-mon[38695]: pgmap v713: 129 pgs: 129 active+clean; 428 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 22 KiB/s rd, 13 MiB/s wr, 595 op/s 2024-05-08T20:23:12.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:11 smithi188 ceph-mon[48761]: pgmap v713: 129 pgs: 129 active+clean; 428 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 22 KiB/s rd, 13 MiB/s wr, 595 op/s 2024-05-08T20:23:12.460 INFO:tasks.workunit.client.1.smithi188.stdout: 10 47887 28.63 MB/sec warmup 96 sec latency 94.674 ms 2024-05-08T20:23:12.720 INFO:tasks.workunit.client.0.smithi083.stdout: 10 80206 46.65 MB/sec warmup 96 sec latency 110.366 ms 2024-05-08T20:23:13.194 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:12 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:23:13.195 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:12 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:23:13.195 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:12 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:23:13.195 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:12 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:23:13.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:12 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:23:13.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:12 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:23:13.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:12 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:23:13.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:12 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:23:13.460 INFO:tasks.workunit.client.1.smithi188.stdout: 10 48341 28.41 MB/sec warmup 97 sec latency 81.249 ms 2024-05-08T20:23:13.724 INFO:tasks.workunit.client.0.smithi083.stdout: 10 81399 47.02 MB/sec warmup 97 sec latency 119.660 ms 2024-05-08T20:23:14.194 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:13 smithi083 ceph-mon[38695]: pgmap v714: 129 pgs: 129 active+clean; 430 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 437 op/s 2024-05-08T20:23:14.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:13 smithi188 ceph-mon[48761]: pgmap v714: 129 pgs: 129 active+clean; 430 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 437 op/s 2024-05-08T20:23:14.460 INFO:tasks.workunit.client.1.smithi188.stdout: 10 48682 28.34 MB/sec warmup 98 sec latency 386.870 ms 2024-05-08T20:23:14.724 INFO:tasks.workunit.client.0.smithi083.stdout: 10 82369 47.05 MB/sec warmup 98 sec latency 234.181 ms 2024-05-08T20:23:15.460 INFO:tasks.workunit.client.1.smithi188.stdout: 10 49435 28.67 MB/sec warmup 99 sec latency 61.810 ms 2024-05-08T20:23:15.724 INFO:tasks.workunit.client.0.smithi083.stdout: 10 82527 46.62 MB/sec warmup 99 sec latency 292.368 ms 2024-05-08T20:23:16.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:15 smithi188 ceph-mon[48761]: pgmap v715: 129 pgs: 129 active+clean; 420 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 14 KiB/s rd, 12 MiB/s wr, 506 op/s 2024-05-08T20:23:16.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:15 smithi083 ceph-mon[38695]: pgmap v715: 129 pgs: 129 active+clean; 420 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 14 KiB/s rd, 12 MiB/s wr, 506 op/s 2024-05-08T20:23:16.460 INFO:tasks.workunit.client.1.smithi188.stdout: 10 49859 28.77 MB/sec warmup 100 sec latency 89.732 ms 2024-05-08T20:23:16.724 INFO:tasks.workunit.client.0.smithi083.stdout: 10 83755 46.69 MB/sec warmup 100 sec latency 135.430 ms 2024-05-08T20:23:17.460 INFO:tasks.workunit.client.1.smithi188.stdout: 10 50248 28.59 MB/sec warmup 101 sec latency 20.698 ms 2024-05-08T20:23:17.724 INFO:tasks.workunit.client.0.smithi083.stdout: 10 84881 46.99 MB/sec warmup 101 sec latency 134.214 ms 2024-05-08T20:23:18.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:17 smithi188 ceph-mon[48761]: pgmap v716: 129 pgs: 129 active+clean; 417 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 22 KiB/s rd, 11 MiB/s wr, 631 op/s 2024-05-08T20:23:18.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:17 smithi083 ceph-mon[38695]: pgmap v716: 129 pgs: 129 active+clean; 417 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 22 KiB/s rd, 11 MiB/s wr, 631 op/s 2024-05-08T20:23:18.460 INFO:tasks.workunit.client.1.smithi188.stdout: 10 50641 28.40 MB/sec warmup 102 sec latency 80.318 ms 2024-05-08T20:23:18.724 INFO:tasks.workunit.client.0.smithi083.stdout: 10 85910 47.08 MB/sec warmup 102 sec latency 194.678 ms 2024-05-08T20:23:19.462 INFO:tasks.workunit.client.1.smithi188.stdout: 10 51075 28.35 MB/sec warmup 103 sec latency 84.561 ms 2024-05-08T20:23:19.724 INFO:tasks.workunit.client.0.smithi083.stdout: 10 86081 46.66 MB/sec warmup 103 sec latency 117.204 ms 2024-05-08T20:23:20.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:19 smithi188 ceph-mon[48761]: pgmap v717: 129 pgs: 129 active+clean; 430 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 741 op/s 2024-05-08T20:23:20.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:19 smithi083 ceph-mon[38695]: pgmap v717: 129 pgs: 129 active+clean; 430 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 741 op/s 2024-05-08T20:23:20.462 INFO:tasks.workunit.client.1.smithi188.stdout: 10 51474 28.27 MB/sec warmup 104 sec latency 142.833 ms 2024-05-08T20:23:20.724 INFO:tasks.workunit.client.0.smithi083.stdout: 10 86941 46.65 MB/sec warmup 104 sec latency 288.770 ms 2024-05-08T20:23:21.462 INFO:tasks.workunit.client.1.smithi188.stdout: 10 52005 28.13 MB/sec warmup 105 sec latency 60.506 ms 2024-05-08T20:23:21.724 INFO:tasks.workunit.client.0.smithi083.stdout: 10 88271 46.94 MB/sec warmup 105 sec latency 157.433 ms 2024-05-08T20:23:22.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:21 smithi188 ceph-mon[48761]: pgmap v718: 129 pgs: 129 active+clean; 429 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 21 KiB/s rd, 13 MiB/s wr, 720 op/s 2024-05-08T20:23:22.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:21 smithi083 ceph-mon[38695]: pgmap v718: 129 pgs: 129 active+clean; 429 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 21 KiB/s rd, 13 MiB/s wr, 720 op/s 2024-05-08T20:23:22.462 INFO:tasks.workunit.client.1.smithi188.stdout: 10 52647 28.38 MB/sec warmup 106 sec latency 52.560 ms 2024-05-08T20:23:22.724 INFO:tasks.workunit.client.0.smithi083.stdout: 10 89436 47.10 MB/sec warmup 106 sec latency 125.701 ms 2024-05-08T20:23:23.462 INFO:tasks.workunit.client.1.smithi188.stdout: 10 53220 28.55 MB/sec warmup 107 sec latency 51.958 ms 2024-05-08T20:23:23.724 INFO:tasks.workunit.client.0.smithi083.stdout: 10 89617 46.70 MB/sec warmup 107 sec latency 123.531 ms 2024-05-08T20:23:24.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:23 smithi188 ceph-mon[48761]: pgmap v719: 129 pgs: 129 active+clean; 427 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 16 KiB/s rd, 9.8 MiB/s wr, 623 op/s 2024-05-08T20:23:24.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:23:24.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:23 smithi083 ceph-mon[38695]: pgmap v719: 129 pgs: 129 active+clean; 427 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 16 KiB/s rd, 9.8 MiB/s wr, 623 op/s 2024-05-08T20:23:24.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:23:24.462 INFO:tasks.workunit.client.1.smithi188.stdout: 10 53615 28.49 MB/sec warmup 108 sec latency 70.542 ms 2024-05-08T20:23:24.724 INFO:tasks.workunit.client.0.smithi083.stdout: 10 91042 46.84 MB/sec warmup 108 sec latency 129.092 ms 2024-05-08T20:23:25.462 INFO:tasks.workunit.client.1.smithi188.stdout: 10 54036 28.29 MB/sec warmup 109 sec latency 75.487 ms 2024-05-08T20:23:25.724 INFO:tasks.workunit.client.0.smithi083.stdout: 10 92038 47.10 MB/sec warmup 109 sec latency 140.343 ms 2024-05-08T20:23:26.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:25 smithi188 ceph-mon[48761]: pgmap v720: 129 pgs: 129 active+clean; 427 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 16 KiB/s rd, 12 MiB/s wr, 683 op/s 2024-05-08T20:23:26.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:25 smithi083 ceph-mon[38695]: pgmap v720: 129 pgs: 129 active+clean; 427 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 16 KiB/s rd, 12 MiB/s wr, 683 op/s 2024-05-08T20:23:26.462 INFO:tasks.workunit.client.1.smithi188.stdout: 10 54437 28.21 MB/sec warmup 110 sec latency 59.927 ms 2024-05-08T20:23:26.724 INFO:tasks.workunit.client.0.smithi083.stdout: 10 93005 47.13 MB/sec warmup 110 sec latency 247.724 ms 2024-05-08T20:23:27.462 INFO:tasks.workunit.client.1.smithi188.stdout: 10 54832 28.19 MB/sec warmup 111 sec latency 85.354 ms 2024-05-08T20:23:27.725 INFO:tasks.workunit.client.0.smithi083.stdout: 10 93175 46.75 MB/sec warmup 111 sec latency 120.185 ms 2024-05-08T20:23:28.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:27 smithi188 ceph-mon[48761]: pgmap v721: 129 pgs: 129 active+clean; 432 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 20 KiB/s rd, 13 MiB/s wr, 713 op/s 2024-05-08T20:23:28.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:27 smithi083 ceph-mon[38695]: pgmap v721: 129 pgs: 129 active+clean; 432 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 20 KiB/s rd, 13 MiB/s wr, 713 op/s 2024-05-08T20:23:28.462 INFO:tasks.workunit.client.1.smithi188.stdout: 10 55182 27.98 MB/sec warmup 112 sec latency 133.574 ms 2024-05-08T20:23:28.725 INFO:tasks.workunit.client.0.smithi083.stdout: 10 94587 46.87 MB/sec warmup 112 sec latency 127.335 ms 2024-05-08T20:23:29.462 INFO:tasks.workunit.client.1.smithi188.stdout: 10 55725 27.93 MB/sec warmup 113 sec latency 76.482 ms 2024-05-08T20:23:29.725 INFO:tasks.workunit.client.0.smithi083.stdout: 10 95664 47.19 MB/sec warmup 113 sec latency 150.480 ms 2024-05-08T20:23:30.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:29 smithi188 ceph-mon[48761]: pgmap v722: 129 pgs: 129 active+clean; 428 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 595 op/s 2024-05-08T20:23:30.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:29 smithi083 ceph-mon[38695]: pgmap v722: 129 pgs: 129 active+clean; 428 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 595 op/s 2024-05-08T20:23:30.462 INFO:tasks.workunit.client.1.smithi188.stdout: 10 56520 28.26 MB/sec warmup 114 sec latency 52.343 ms 2024-05-08T20:23:30.725 INFO:tasks.workunit.client.0.smithi083.stdout: 10 96559 47.16 MB/sec warmup 114 sec latency 127.159 ms 2024-05-08T20:23:31.462 INFO:tasks.workunit.client.1.smithi188.stdout: 10 57040 28.38 MB/sec warmup 115 sec latency 45.364 ms 2024-05-08T20:23:31.725 INFO:tasks.workunit.client.0.smithi083.stdout: 10 96775 46.79 MB/sec warmup 115 sec latency 110.231 ms 2024-05-08T20:23:32.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:31 smithi188 ceph-mon[48761]: pgmap v723: 129 pgs: 129 active+clean; 417 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 554 op/s 2024-05-08T20:23:32.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:31 smithi083 ceph-mon[38695]: pgmap v723: 129 pgs: 129 active+clean; 417 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 554 op/s 2024-05-08T20:23:32.462 INFO:tasks.workunit.client.1.smithi188.stdout: 10 57469 28.21 MB/sec warmup 116 sec latency 64.336 ms 2024-05-08T20:23:32.725 INFO:tasks.workunit.client.0.smithi083.stdout: 10 98322 47.01 MB/sec warmup 116 sec latency 121.291 ms 2024-05-08T20:23:33.463 INFO:tasks.workunit.client.1.smithi188.stdout: 10 57945 28.12 MB/sec warmup 117 sec latency 74.136 ms 2024-05-08T20:23:33.725 INFO:tasks.workunit.client.0.smithi083.stdout: 10 100028 47.57 MB/sec warmup 117 sec latency 126.824 ms 2024-05-08T20:23:34.344 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:34 smithi188 ceph-mon[48761]: pgmap v724: 129 pgs: 129 active+clean; 417 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 9.1 MiB/s wr, 449 op/s 2024-05-08T20:23:34.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:34 smithi083 ceph-mon[38695]: pgmap v724: 129 pgs: 129 active+clean; 417 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 9.1 MiB/s wr, 449 op/s 2024-05-08T20:23:34.544 INFO:tasks.workunit.client.1.smithi188.stdout: 10 58377 28.14 MB/sec warmup 118 sec latency 101.990 ms 2024-05-08T20:23:34.725 INFO:tasks.workunit.client.0.smithi083.stdout: 10 100165 47.20 MB/sec warmup 118 sec latency 110.357 ms 2024-05-08T20:23:35.463 INFO:tasks.workunit.client.1.smithi188.stdout: 10 58744 27.94 MB/sec warmup 119 sec latency 105.249 ms 2024-05-08T20:23:35.725 INFO:tasks.workunit.client.0.smithi083.stdout: 10 100985 47.19 MB/sec warmup 119 sec latency 155.320 ms 2024-05-08T20:23:36.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:36 smithi188 ceph-mon[48761]: pgmap v725: 129 pgs: 129 active+clean; 433 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 578 op/s 2024-05-08T20:23:36.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:36 smithi083 ceph-mon[38695]: pgmap v725: 129 pgs: 129 active+clean; 433 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 578 op/s 2024-05-08T20:23:37.463 INFO:tasks.workunit.client.1.smithi188.stdout: 10 59812 51.56 MB/sec execute 1 sec latency 62.891 ms 2024-05-08T20:23:37.725 INFO:tasks.workunit.client.0.smithi083.stdout: 10 103593 69.42 MB/sec execute 1 sec latency 133.991 ms 2024-05-08T20:23:38.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:38 smithi188 ceph-mon[48761]: pgmap v726: 129 pgs: 129 active+clean; 435 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 615 op/s 2024-05-08T20:23:38.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:38 smithi083 ceph-mon[38695]: pgmap v726: 129 pgs: 129 active+clean; 435 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 615 op/s 2024-05-08T20:23:38.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 60253 40.92 MB/sec execute 2 sec latency 53.357 ms 2024-05-08T20:23:38.725 INFO:tasks.workunit.client.0.smithi083.stdout: 10 103716 36.19 MB/sec execute 2 sec latency 126.400 ms 2024-05-08T20:23:39.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:23:39.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:23:39.463 INFO:tasks.workunit.client.1.smithi188.stdout: 10 60618 37.18 MB/sec execute 3 sec latency 81.667 ms 2024-05-08T20:23:39.725 INFO:tasks.workunit.client.0.smithi083.stdout: 10 104599 39.94 MB/sec execute 3 sec latency 124.345 ms 2024-05-08T20:23:40.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:40 smithi188 ceph-mon[48761]: pgmap v727: 129 pgs: 129 active+clean; 426 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 15 KiB/s rd, 11 MiB/s wr, 576 op/s 2024-05-08T20:23:40.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:40 smithi083 ceph-mon[38695]: pgmap v727: 129 pgs: 129 active+clean; 426 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 15 KiB/s rd, 11 MiB/s wr, 576 op/s 2024-05-08T20:23:40.463 INFO:tasks.workunit.client.1.smithi188.stdout: 10 61065 29.73 MB/sec execute 4 sec latency 20.562 ms 2024-05-08T20:23:40.725 INFO:tasks.workunit.client.0.smithi083.stdout: 10 105764 46.25 MB/sec execute 4 sec latency 155.137 ms 2024-05-08T20:23:41.463 INFO:tasks.workunit.client.1.smithi188.stdout: 10 61511 27.97 MB/sec execute 5 sec latency 74.429 ms 2024-05-08T20:23:41.729 INFO:tasks.workunit.client.0.smithi083.stdout: 10 106222 42.27 MB/sec execute 5 sec latency 471.474 ms 2024-05-08T20:23:42.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:42 smithi188 ceph-mon[48761]: pgmap v728: 129 pgs: 129 active+clean; 432 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 23 KiB/s rd, 13 MiB/s wr, 569 op/s 2024-05-08T20:23:42.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:42 smithi083 ceph-mon[38695]: pgmap v728: 129 pgs: 129 active+clean; 432 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 23 KiB/s rd, 13 MiB/s wr, 569 op/s 2024-05-08T20:23:42.463 INFO:tasks.workunit.client.1.smithi188.stdout: 10 61983 27.85 MB/sec execute 6 sec latency 99.873 ms 2024-05-08T20:23:42.727 INFO:tasks.workunit.client.0.smithi083.stdout: 10 106237 35.23 MB/sec execute 6 sec latency 880.126 ms 2024-05-08T20:23:43.463 INFO:tasks.workunit.client.1.smithi188.stdout: 10 62423 24.72 MB/sec execute 7 sec latency 81.258 ms 2024-05-08T20:23:43.727 INFO:tasks.workunit.client.0.smithi083.stdout: 10 106686 33.70 MB/sec execute 7 sec latency 957.944 ms 2024-05-08T20:23:44.325 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:44 smithi083 ceph-mon[38695]: pgmap v729: 129 pgs: 129 active+clean; 434 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 450 op/s 2024-05-08T20:23:44.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:44 smithi188 ceph-mon[48761]: pgmap v729: 129 pgs: 129 active+clean; 434 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 450 op/s 2024-05-08T20:23:44.463 INFO:tasks.workunit.client.1.smithi188.stdout: 10 62936 25.14 MB/sec execute 8 sec latency 86.092 ms 2024-05-08T20:23:44.727 INFO:tasks.workunit.client.0.smithi083.stdout: 10 107130 32.56 MB/sec execute 8 sec latency 721.960 ms 2024-05-08T20:23:45.463 INFO:tasks.workunit.client.1.smithi188.stdout: 10 63660 29.01 MB/sec execute 9 sec latency 65.561 ms 2024-05-08T20:23:45.728 INFO:tasks.workunit.client.0.smithi083.stdout: 10 107135 28.95 MB/sec execute 9 sec latency 440.427 ms 2024-05-08T20:23:46.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:46 smithi188 ceph-mon[48761]: pgmap v730: 129 pgs: 129 active+clean; 421 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 14 KiB/s rd, 13 MiB/s wr, 544 op/s 2024-05-08T20:23:46.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:46 smithi083 ceph-mon[38695]: pgmap v730: 129 pgs: 129 active+clean; 421 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 14 KiB/s rd, 13 MiB/s wr, 544 op/s 2024-05-08T20:23:46.463 INFO:tasks.workunit.client.1.smithi188.stdout: 10 64107 29.98 MB/sec execute 10 sec latency 81.168 ms 2024-05-08T20:23:46.728 INFO:tasks.workunit.client.0.smithi083.stdout: 10 107184 26.18 MB/sec execute 10 sec latency 418.202 ms 2024-05-08T20:23:47.463 INFO:tasks.workunit.client.1.smithi188.stdout: 10 64473 27.92 MB/sec execute 11 sec latency 69.837 ms 2024-05-08T20:23:47.729 INFO:tasks.workunit.client.0.smithi083.stdout: 10 107388 24.25 MB/sec execute 11 sec latency 100.696 ms 2024-05-08T20:23:48.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:48 smithi188 ceph-mon[48761]: pgmap v731: 129 pgs: 129 active+clean; 418 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 22 KiB/s rd, 12 MiB/s wr, 546 op/s 2024-05-08T20:23:48.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:48 smithi083 ceph-mon[38695]: pgmap v731: 129 pgs: 129 active+clean; 418 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 22 KiB/s rd, 12 MiB/s wr, 546 op/s 2024-05-08T20:23:48.463 INFO:tasks.workunit.client.1.smithi188.stdout: 10 64893 26.79 MB/sec execute 12 sec latency 27.585 ms 2024-05-08T20:23:48.729 INFO:tasks.workunit.client.0.smithi083.stdout: 10 108844 27.42 MB/sec execute 12 sec latency 151.252 ms 2024-05-08T20:23:49.463 INFO:tasks.workunit.client.1.smithi188.stdout: 10 65342 27.03 MB/sec execute 13 sec latency 111.027 ms 2024-05-08T20:23:49.729 INFO:tasks.workunit.client.0.smithi083.stdout: 10 109781 30.97 MB/sec execute 13 sec latency 141.583 ms 2024-05-08T20:23:50.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:50 smithi188 ceph-mon[48761]: pgmap v732: 129 pgs: 129 active+clean; 424 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 519 op/s 2024-05-08T20:23:50.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:50 smithi083 ceph-mon[38695]: pgmap v732: 129 pgs: 129 active+clean; 424 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 519 op/s 2024-05-08T20:23:50.463 INFO:tasks.workunit.client.1.smithi188.stdout: 10 65716 25.79 MB/sec execute 14 sec latency 102.653 ms 2024-05-08T20:23:50.729 INFO:tasks.workunit.client.0.smithi083.stdout: 10 110743 32.36 MB/sec execute 14 sec latency 118.211 ms 2024-05-08T20:23:51.463 INFO:tasks.workunit.client.1.smithi188.stdout: 10 66261 25.29 MB/sec execute 15 sec latency 79.183 ms 2024-05-08T20:23:51.729 INFO:tasks.workunit.client.0.smithi083.stdout: 10 111227 31.98 MB/sec execute 15 sec latency 118.418 ms 2024-05-08T20:23:52.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:52 smithi188 ceph-mon[48761]: pgmap v733: 129 pgs: 129 active+clean; 432 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 22 KiB/s rd, 14 MiB/s wr, 620 op/s 2024-05-08T20:23:52.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:52 smithi083 ceph-mon[38695]: pgmap v733: 129 pgs: 129 active+clean; 432 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 22 KiB/s rd, 14 MiB/s wr, 620 op/s 2024-05-08T20:23:52.463 INFO:tasks.workunit.client.1.smithi188.stdout: 10 66910 27.17 MB/sec execute 16 sec latency 58.239 ms 2024-05-08T20:23:52.729 INFO:tasks.workunit.client.0.smithi083.stdout: 10 112307 32.35 MB/sec execute 16 sec latency 282.547 ms 2024-05-08T20:23:53.463 INFO:tasks.workunit.client.1.smithi188.stdout: 10 67419 27.61 MB/sec execute 17 sec latency 49.304 ms 2024-05-08T20:23:53.729 INFO:tasks.workunit.client.0.smithi083.stdout: 10 113326 34.93 MB/sec execute 17 sec latency 135.622 ms 2024-05-08T20:23:54.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:54 smithi188 ceph-mon[48761]: pgmap v734: 129 pgs: 129 active+clean; 429 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 517 op/s 2024-05-08T20:23:54.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:23:54.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:54 smithi083 ceph-mon[38695]: pgmap v734: 129 pgs: 129 active+clean; 429 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 517 op/s 2024-05-08T20:23:54.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:23:54.463 INFO:tasks.workunit.client.1.smithi188.stdout: 10 67819 27.56 MB/sec execute 18 sec latency 83.707 ms 2024-05-08T20:23:54.729 INFO:tasks.workunit.client.0.smithi083.stdout: 10 114285 35.79 MB/sec execute 18 sec latency 128.809 ms 2024-05-08T20:23:55.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 68205 26.43 MB/sec execute 19 sec latency 78.549 ms 2024-05-08T20:23:55.729 INFO:tasks.workunit.client.0.smithi083.stdout: 10 114527 34.21 MB/sec execute 19 sec latency 123.620 ms 2024-05-08T20:23:56.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:56 smithi188 ceph-mon[48761]: pgmap v735: 129 pgs: 129 active+clean; 433 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 14 KiB/s rd, 12 MiB/s wr, 565 op/s 2024-05-08T20:23:56.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:56 smithi083 ceph-mon[38695]: pgmap v735: 129 pgs: 129 active+clean; 433 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 14 KiB/s rd, 12 MiB/s wr, 565 op/s 2024-05-08T20:23:56.463 INFO:tasks.workunit.client.1.smithi188.stdout: 10 68638 26.14 MB/sec execute 20 sec latency 64.990 ms 2024-05-08T20:23:56.732 INFO:tasks.workunit.client.0.smithi083.stdout: 10 116073 36.20 MB/sec execute 20 sec latency 121.797 ms 2024-05-08T20:23:57.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 69097 26.18 MB/sec execute 21 sec latency 92.399 ms 2024-05-08T20:23:57.733 INFO:tasks.workunit.client.0.smithi083.stdout: 10 117321 38.54 MB/sec execute 21 sec latency 222.644 ms 2024-05-08T20:23:58.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:23:58 smithi188 ceph-mon[48761]: pgmap v736: 129 pgs: 129 active+clean; 430 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 21 KiB/s rd, 13 MiB/s wr, 576 op/s 2024-05-08T20:23:58.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:23:58 smithi083 ceph-mon[38695]: pgmap v736: 129 pgs: 129 active+clean; 430 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 21 KiB/s rd, 13 MiB/s wr, 576 op/s 2024-05-08T20:23:58.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 69519 25.30 MB/sec execute 22 sec latency 115.689 ms 2024-05-08T20:23:58.733 INFO:tasks.workunit.client.0.smithi083.stdout: 10 117862 38.00 MB/sec execute 22 sec latency 270.704 ms 2024-05-08T20:23:59.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 70064 25.66 MB/sec execute 23 sec latency 82.365 ms 2024-05-08T20:23:59.733 INFO:tasks.workunit.client.0.smithi083.stdout: 10 118359 37.49 MB/sec execute 23 sec latency 124.212 ms 2024-05-08T20:24:00.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:00 smithi188 ceph-mon[48761]: pgmap v737: 129 pgs: 129 active+clean; 429 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 516 op/s 2024-05-08T20:24:00.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:00 smithi083 ceph-mon[38695]: pgmap v737: 129 pgs: 129 active+clean; 429 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 516 op/s 2024-05-08T20:24:00.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 70755 26.82 MB/sec execute 24 sec latency 64.627 ms 2024-05-08T20:24:00.733 INFO:tasks.workunit.client.0.smithi083.stdout: 10 119738 38.67 MB/sec execute 24 sec latency 135.764 ms 2024-05-08T20:24:01.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 71209 27.27 MB/sec execute 25 sec latency 55.027 ms 2024-05-08T20:24:01.733 INFO:tasks.workunit.client.0.smithi083.stdout: 10 121316 41.00 MB/sec execute 25 sec latency 114.341 ms 2024-05-08T20:24:02.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:02 smithi188 ceph-mon[48761]: pgmap v738: 129 pgs: 129 active+clean; 420 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 22 KiB/s rd, 13 MiB/s wr, 612 op/s 2024-05-08T20:24:02.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:02 smithi083 ceph-mon[38695]: pgmap v738: 129 pgs: 129 active+clean; 420 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 22 KiB/s rd, 13 MiB/s wr, 612 op/s 2024-05-08T20:24:02.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 71573 26.54 MB/sec execute 26 sec latency 74.126 ms 2024-05-08T20:24:02.733 INFO:tasks.workunit.client.0.smithi083.stdout: 10 121441 39.54 MB/sec execute 26 sec latency 116.935 ms 2024-05-08T20:24:03.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 71919 25.99 MB/sec execute 27 sec latency 84.873 ms 2024-05-08T20:24:03.733 INFO:tasks.workunit.client.0.smithi083.stdout: 10 122322 39.83 MB/sec execute 27 sec latency 128.405 ms 2024-05-08T20:24:04.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:04 smithi188 ceph-mon[48761]: pgmap v739: 129 pgs: 129 active+clean; 421 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 16 KiB/s rd, 9.5 MiB/s wr, 452 op/s 2024-05-08T20:24:04.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:04 smithi083 ceph-mon[38695]: pgmap v739: 129 pgs: 129 active+clean; 421 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 16 KiB/s rd, 9.5 MiB/s wr, 452 op/s 2024-05-08T20:24:04.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 72266 25.76 MB/sec execute 28 sec latency 106.062 ms 2024-05-08T20:24:04.733 INFO:tasks.workunit.client.0.smithi083.stdout: 10 123659 41.08 MB/sec execute 28 sec latency 137.068 ms 2024-05-08T20:24:05.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 72663 25.56 MB/sec execute 29 sec latency 143.696 ms 2024-05-08T20:24:05.733 INFO:tasks.workunit.client.0.smithi083.stdout: 10 124911 41.98 MB/sec execute 29 sec latency 127.047 ms 2024-05-08T20:24:06.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:06 smithi188 ceph-mon[48761]: pgmap v740: 129 pgs: 129 active+clean; 431 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 16 KiB/s rd, 12 MiB/s wr, 525 op/s 2024-05-08T20:24:06.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:06 smithi083 ceph-mon[38695]: pgmap v740: 129 pgs: 129 active+clean; 431 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 16 KiB/s rd, 12 MiB/s wr, 525 op/s 2024-05-08T20:24:06.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 73169 25.01 MB/sec execute 30 sec latency 82.494 ms 2024-05-08T20:24:06.733 INFO:tasks.workunit.client.0.smithi083.stdout: 10 125030 40.67 MB/sec execute 30 sec latency 161.915 ms 2024-05-08T20:24:07.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 73708 25.42 MB/sec execute 31 sec latency 62.271 ms 2024-05-08T20:24:07.734 INFO:tasks.workunit.client.0.smithi083.stdout: 10 126170 41.01 MB/sec execute 31 sec latency 116.956 ms 2024-05-08T20:24:08.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:08 smithi188 ceph-mon[48761]: pgmap v741: 129 pgs: 129 active+clean; 432 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 22 KiB/s rd, 12 MiB/s wr, 571 op/s 2024-05-08T20:24:08.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:08 smithi083 ceph-mon[38695]: pgmap v741: 129 pgs: 129 active+clean; 432 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 22 KiB/s rd, 12 MiB/s wr, 571 op/s 2024-05-08T20:24:08.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 74337 26.14 MB/sec execute 32 sec latency 57.123 ms 2024-05-08T20:24:08.734 INFO:tasks.workunit.client.0.smithi083.stdout: 10 127380 42.15 MB/sec execute 32 sec latency 122.857 ms 2024-05-08T20:24:09.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:24:09.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:24:09.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 74719 26.41 MB/sec execute 33 sec latency 74.677 ms 2024-05-08T20:24:09.735 INFO:tasks.workunit.client.0.smithi083.stdout: 10 128462 42.69 MB/sec execute 33 sec latency 131.503 ms 2024-05-08T20:24:10.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:10 smithi188 ceph-mon[48761]: pgmap v742: 129 pgs: 129 active+clean; 433 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 15 KiB/s rd, 11 MiB/s wr, 570 op/s 2024-05-08T20:24:10.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:10 smithi083 ceph-mon[38695]: pgmap v742: 129 pgs: 129 active+clean; 433 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 15 KiB/s rd, 11 MiB/s wr, 570 op/s 2024-05-08T20:24:10.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 75066 25.92 MB/sec execute 34 sec latency 81.429 ms 2024-05-08T20:24:10.735 INFO:tasks.workunit.client.0.smithi083.stdout: 10 128632 41.56 MB/sec execute 34 sec latency 118.888 ms 2024-05-08T20:24:11.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 75470 25.48 MB/sec execute 35 sec latency 88.792 ms 2024-05-08T20:24:11.735 INFO:tasks.workunit.client.0.smithi083.stdout: 10 130034 42.09 MB/sec execute 35 sec latency 134.786 ms 2024-05-08T20:24:12.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:12 smithi188 ceph-mon[48761]: pgmap v743: 129 pgs: 129 active+clean; 441 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 22 KiB/s rd, 13 MiB/s wr, 599 op/s 2024-05-08T20:24:12.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:12 smithi083 ceph-mon[38695]: pgmap v743: 129 pgs: 129 active+clean; 441 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 22 KiB/s rd, 13 MiB/s wr, 599 op/s 2024-05-08T20:24:12.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 75909 25.45 MB/sec execute 36 sec latency 104.134 ms 2024-05-08T20:24:12.736 INFO:tasks.workunit.client.0.smithi083.stdout: 10 131235 43.31 MB/sec execute 36 sec latency 128.629 ms 2024-05-08T20:24:13.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:13 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:24:13.444 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:13 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:24:13.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 76280 25.23 MB/sec execute 37 sec latency 145.887 ms 2024-05-08T20:24:13.740 INFO:tasks.workunit.client.0.smithi083.stdout: 10 132012 43.24 MB/sec execute 37 sec latency 285.242 ms 2024-05-08T20:24:14.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:14 smithi188 ceph-mon[48761]: pgmap v744: 129 pgs: 129 active+clean; 442 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 431 op/s 2024-05-08T20:24:14.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:14 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:24:14.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:14 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:24:14.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:14 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:24:14.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:14 smithi083 ceph-mon[38695]: pgmap v744: 129 pgs: 129 active+clean; 442 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 431 op/s 2024-05-08T20:24:14.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:14 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:24:14.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:14 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:24:14.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:14 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:24:14.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 76733 24.83 MB/sec execute 38 sec latency 87.557 ms 2024-05-08T20:24:14.740 INFO:tasks.workunit.client.0.smithi083.stdout: 10 132193 42.22 MB/sec execute 38 sec latency 109.609 ms 2024-05-08T20:24:15.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 77277 25.10 MB/sec execute 39 sec latency 85.461 ms 2024-05-08T20:24:15.740 INFO:tasks.workunit.client.0.smithi083.stdout: 10 133403 42.52 MB/sec execute 39 sec latency 147.613 ms 2024-05-08T20:24:16.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:16 smithi188 ceph-mon[48761]: pgmap v745: 129 pgs: 129 active+clean; 446 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 581 op/s 2024-05-08T20:24:16.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:16 smithi083 ceph-mon[38695]: pgmap v745: 129 pgs: 129 active+clean; 446 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 581 op/s 2024-05-08T20:24:16.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 77928 25.80 MB/sec execute 40 sec latency 47.764 ms 2024-05-08T20:24:16.740 INFO:tasks.workunit.client.0.smithi083.stdout: 10 134568 43.49 MB/sec execute 40 sec latency 141.740 ms 2024-05-08T20:24:17.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 78355 25.96 MB/sec execute 41 sec latency 55.198 ms 2024-05-08T20:24:17.740 INFO:tasks.workunit.client.0.smithi083.stdout: 10 135559 43.69 MB/sec execute 41 sec latency 125.295 ms 2024-05-08T20:24:18.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:18 smithi188 ceph-mon[48761]: pgmap v746: 129 pgs: 129 active+clean; 426 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 591 op/s 2024-05-08T20:24:18.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:18 smithi083 ceph-mon[38695]: pgmap v746: 129 pgs: 129 active+clean; 426 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 591 op/s 2024-05-08T20:24:18.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 78734 25.59 MB/sec execute 42 sec latency 43.959 ms 2024-05-08T20:24:18.742 INFO:tasks.workunit.client.0.smithi083.stdout: 10 135740 42.75 MB/sec execute 42 sec latency 117.388 ms 2024-05-08T20:24:19.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 79188 25.38 MB/sec execute 43 sec latency 87.331 ms 2024-05-08T20:24:19.742 INFO:tasks.workunit.client.0.smithi083.stdout: 10 136443 42.78 MB/sec execute 43 sec latency 435.196 ms 2024-05-08T20:24:20.195 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:20 smithi083 ceph-mon[38695]: pgmap v747: 129 pgs: 129 active+clean; 427 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 15 KiB/s rd, 10 MiB/s wr, 533 op/s 2024-05-08T20:24:20.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:20 smithi188 ceph-mon[48761]: pgmap v747: 129 pgs: 129 active+clean; 427 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 15 KiB/s rd, 10 MiB/s wr, 533 op/s 2024-05-08T20:24:20.464 INFO:tasks.workunit.client.1.smithi188.stdout: 10 79621 25.39 MB/sec execute 44 sec latency 86.792 ms 2024-05-08T20:24:20.742 INFO:tasks.workunit.client.0.smithi083.stdout: 10 136585 41.84 MB/sec execute 44 sec latency 741.701 ms 2024-05-08T20:24:21.466 INFO:tasks.workunit.client.1.smithi188.stdout: 10 80042 25.08 MB/sec execute 45 sec latency 110.989 ms 2024-05-08T20:24:21.742 INFO:tasks.workunit.client.0.smithi083.stdout: 10 136712 40.93 MB/sec execute 45 sec latency 624.216 ms 2024-05-08T20:24:22.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:22 smithi188 ceph-mon[48761]: pgmap v748: 129 pgs: 129 active+clean; 446 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 20 KiB/s rd, 13 MiB/s wr, 612 op/s 2024-05-08T20:24:22.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:22 smithi083 ceph-mon[38695]: pgmap v748: 129 pgs: 129 active+clean; 446 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 20 KiB/s rd, 13 MiB/s wr, 612 op/s 2024-05-08T20:24:22.466 INFO:tasks.workunit.client.1.smithi188.stdout: 10 80570 25.04 MB/sec execute 46 sec latency 66.657 ms 2024-05-08T20:24:22.742 INFO:tasks.workunit.client.0.smithi083.stdout: 10 137145 40.35 MB/sec execute 46 sec latency 464.846 ms 2024-05-08T20:24:23.466 INFO:tasks.workunit.client.1.smithi188.stdout: 10 81244 25.73 MB/sec execute 47 sec latency 61.213 ms 2024-05-08T20:24:23.742 INFO:tasks.workunit.client.0.smithi083.stdout: 10 137350 39.81 MB/sec execute 47 sec latency 426.650 ms 2024-05-08T20:24:24.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:24 smithi083 ceph-mon[38695]: pgmap v749: 129 pgs: 129 active+clean; 445 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 509 op/s 2024-05-08T20:24:24.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:24:24.466 INFO:tasks.workunit.client.1.smithi188.stdout: 10 81692 25.86 MB/sec execute 48 sec latency 63.470 ms 2024-05-08T20:24:24.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:24 smithi188 ceph-mon[48761]: pgmap v749: 129 pgs: 129 active+clean; 445 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 509 op/s 2024-05-08T20:24:24.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:24 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:24:24.742 INFO:tasks.workunit.client.0.smithi083.stdout: 10 137778 39.86 MB/sec execute 48 sec latency 345.831 ms 2024-05-08T20:24:25.466 INFO:tasks.workunit.client.1.smithi188.stdout: 10 82073 25.77 MB/sec execute 49 sec latency 98.140 ms 2024-05-08T20:24:25.742 INFO:tasks.workunit.client.0.smithi083.stdout: 10 139052 40.43 MB/sec execute 49 sec latency 368.351 ms 2024-05-08T20:24:26.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:26 smithi083 ceph-mon[38695]: pgmap v750: 129 pgs: 129 active+clean; 439 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 559 op/s 2024-05-08T20:24:26.466 INFO:tasks.workunit.client.1.smithi188.stdout: 10 82487 25.40 MB/sec execute 50 sec latency 67.434 ms 2024-05-08T20:24:26.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:26 smithi188 ceph-mon[48761]: pgmap v750: 129 pgs: 129 active+clean; 439 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 559 op/s 2024-05-08T20:24:26.742 INFO:tasks.workunit.client.0.smithi083.stdout: 10 139193 39.69 MB/sec execute 50 sec latency 124.589 ms 2024-05-08T20:24:27.466 INFO:tasks.workunit.client.1.smithi188.stdout: 10 82941 25.45 MB/sec execute 51 sec latency 78.688 ms 2024-05-08T20:24:27.742 INFO:tasks.workunit.client.0.smithi083.stdout: 10 140351 39.92 MB/sec execute 51 sec latency 121.851 ms 2024-05-08T20:24:28.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:28 smithi083 ceph-mon[38695]: pgmap v751: 129 pgs: 129 active+clean; 440 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 22 KiB/s rd, 12 MiB/s wr, 558 op/s 2024-05-08T20:24:28.466 INFO:tasks.workunit.client.1.smithi188.stdout: 10 83320 25.29 MB/sec execute 52 sec latency 100.529 ms 2024-05-08T20:24:28.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:28 smithi188 ceph-mon[48761]: pgmap v751: 129 pgs: 129 active+clean; 440 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 22 KiB/s rd, 12 MiB/s wr, 558 op/s 2024-05-08T20:24:28.742 INFO:tasks.workunit.client.0.smithi083.stdout: 10 141674 40.82 MB/sec execute 52 sec latency 124.073 ms 2024-05-08T20:24:29.466 INFO:tasks.workunit.client.1.smithi188.stdout: 10 83734 24.96 MB/sec execute 53 sec latency 127.254 ms 2024-05-08T20:24:29.743 INFO:tasks.workunit.client.0.smithi083.stdout: 10 142652 41.01 MB/sec execute 53 sec latency 109.156 ms 2024-05-08T20:24:30.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:30 smithi083 ceph-mon[38695]: pgmap v752: 129 pgs: 129 active+clean; 437 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 497 op/s 2024-05-08T20:24:30.466 INFO:tasks.workunit.client.1.smithi188.stdout: 10 84276 25.02 MB/sec execute 54 sec latency 69.420 ms 2024-05-08T20:24:30.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:30 smithi188 ceph-mon[48761]: pgmap v752: 129 pgs: 129 active+clean; 437 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 497 op/s 2024-05-08T20:24:30.742 INFO:tasks.workunit.client.0.smithi083.stdout: 10 142850 40.34 MB/sec execute 54 sec latency 250.655 ms 2024-05-08T20:24:31.466 INFO:tasks.workunit.client.1.smithi188.stdout: 10 84971 25.62 MB/sec execute 55 sec latency 53.880 ms 2024-05-08T20:24:31.742 INFO:tasks.workunit.client.0.smithi083.stdout: 10 144324 40.77 MB/sec execute 55 sec latency 133.173 ms 2024-05-08T20:24:32.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:32 smithi083 ceph-mon[38695]: pgmap v753: 129 pgs: 129 active+clean; 431 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 22 KiB/s rd, 13 MiB/s wr, 642 op/s 2024-05-08T20:24:32.466 INFO:tasks.workunit.client.1.smithi188.stdout: 10 85355 25.79 MB/sec execute 56 sec latency 63.570 ms 2024-05-08T20:24:32.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:32 smithi188 ceph-mon[48761]: pgmap v753: 129 pgs: 129 active+clean; 431 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 22 KiB/s rd, 13 MiB/s wr, 642 op/s 2024-05-08T20:24:32.743 INFO:tasks.workunit.client.0.smithi083.stdout: 10 146048 42.11 MB/sec execute 56 sec latency 128.459 ms 2024-05-08T20:24:33.467 INFO:tasks.workunit.client.1.smithi188.stdout: 10 85657 25.51 MB/sec execute 57 sec latency 59.189 ms 2024-05-08T20:24:33.743 INFO:tasks.workunit.client.0.smithi083.stdout: 10 146262 41.51 MB/sec execute 57 sec latency 117.807 ms 2024-05-08T20:24:34.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:34 smithi083 ceph-mon[38695]: pgmap v754: 129 pgs: 129 active+clean; 431 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 18 KiB/s rd, 9.1 MiB/s wr, 460 op/s 2024-05-08T20:24:34.467 INFO:tasks.workunit.client.1.smithi188.stdout: 10 85957 25.17 MB/sec execute 58 sec latency 90.839 ms 2024-05-08T20:24:34.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:34 smithi188 ceph-mon[48761]: pgmap v754: 129 pgs: 129 active+clean; 431 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 18 KiB/s rd, 9.1 MiB/s wr, 460 op/s 2024-05-08T20:24:34.743 INFO:tasks.workunit.client.0.smithi083.stdout: 10 147148 41.62 MB/sec execute 58 sec latency 114.211 ms 2024-05-08T20:24:35.467 INFO:tasks.workunit.client.1.smithi188.stdout: 10 86294 24.98 MB/sec execute 59 sec latency 108.361 ms 2024-05-08T20:24:35.743 INFO:tasks.workunit.client.0.smithi083.stdout: 10 148494 42.18 MB/sec execute 59 sec latency 136.115 ms 2024-05-08T20:24:36.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:36 smithi083 ceph-mon[38695]: pgmap v755: 129 pgs: 129 active+clean; 432 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 493 op/s 2024-05-08T20:24:36.467 INFO:tasks.workunit.client.1.smithi188.stdout: 10 86724 24.95 MB/sec execute 60 sec latency 119.838 ms 2024-05-08T20:24:36.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:36 smithi188 ceph-mon[48761]: pgmap v755: 129 pgs: 129 active+clean; 432 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 493 op/s 2024-05-08T20:24:36.743 INFO:tasks.workunit.client.0.smithi083.stdout: 10 149715 42.59 MB/sec execute 60 sec latency 119.251 ms 2024-05-08T20:24:37.470 INFO:tasks.workunit.client.1.smithi188.stdout: 10 87175 24.79 MB/sec execute 61 sec latency 108.119 ms 2024-05-08T20:24:37.744 INFO:tasks.workunit.client.0.smithi083.stdout: 10 149893 41.96 MB/sec execute 61 sec latency 120.006 ms 2024-05-08T20:24:38.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:38 smithi083 ceph-mon[38695]: pgmap v756: 129 pgs: 129 active+clean; 445 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 22 KiB/s rd, 12 MiB/s wr, 522 op/s 2024-05-08T20:24:38.470 INFO:tasks.workunit.client.1.smithi188.stdout: 10 87756 24.83 MB/sec execute 62 sec latency 61.334 ms 2024-05-08T20:24:38.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:38 smithi188 ceph-mon[48761]: pgmap v756: 129 pgs: 129 active+clean; 445 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 22 KiB/s rd, 12 MiB/s wr, 522 op/s 2024-05-08T20:24:38.744 INFO:tasks.workunit.client.0.smithi083.stdout: 10 150984 42.09 MB/sec execute 62 sec latency 127.957 ms 2024-05-08T20:24:39.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:24:39.470 INFO:tasks.workunit.client.1.smithi188.stdout: 10 88397 25.33 MB/sec execute 63 sec latency 79.346 ms 2024-05-08T20:24:39.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:24:39.744 INFO:tasks.workunit.client.0.smithi083.stdout: 10 152306 42.80 MB/sec execute 63 sec latency 113.114 ms 2024-05-08T20:24:40.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:40 smithi083 ceph-mon[38695]: pgmap v757: 129 pgs: 129 active+clean; 436 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 495 op/s 2024-05-08T20:24:40.453 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:40 smithi188 ceph-mon[48761]: pgmap v757: 129 pgs: 129 active+clean; 436 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 495 op/s 2024-05-08T20:24:40.470 INFO:tasks.workunit.client.1.smithi188.stdout: 10 88866 25.49 MB/sec execute 64 sec latency 66.661 ms 2024-05-08T20:24:40.744 INFO:tasks.workunit.client.0.smithi083.stdout: 10 153291 42.92 MB/sec execute 64 sec latency 109.588 ms 2024-05-08T20:24:41.470 INFO:tasks.workunit.client.1.smithi188.stdout: 10 89311 25.35 MB/sec execute 65 sec latency 83.220 ms 2024-05-08T20:24:41.744 INFO:tasks.workunit.client.0.smithi083.stdout: 10 153515 42.34 MB/sec execute 65 sec latency 136.321 ms 2024-05-08T20:24:42.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:42 smithi083 ceph-mon[38695]: pgmap v758: 129 pgs: 129 active+clean; 432 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 22 KiB/s rd, 12 MiB/s wr, 621 op/s 2024-05-08T20:24:42.471 INFO:tasks.workunit.client.1.smithi188.stdout: 10 89688 25.14 MB/sec execute 66 sec latency 88.709 ms 2024-05-08T20:24:42.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:42 smithi188 ceph-mon[48761]: pgmap v758: 129 pgs: 129 active+clean; 432 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 22 KiB/s rd, 12 MiB/s wr, 621 op/s 2024-05-08T20:24:42.744 INFO:tasks.workunit.client.0.smithi083.stdout: 10 154956 42.66 MB/sec execute 66 sec latency 151.553 ms 2024-05-08T20:24:43.471 INFO:tasks.workunit.client.1.smithi188.stdout: 10 90030 25.02 MB/sec execute 67 sec latency 131.657 ms 2024-05-08T20:24:43.744 INFO:tasks.workunit.client.0.smithi083.stdout: 10 156325 43.47 MB/sec execute 67 sec latency 125.680 ms 2024-05-08T20:24:44.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:44 smithi083 ceph-mon[38695]: pgmap v759: 129 pgs: 129 active+clean; 433 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 12 KiB/s rd, 9.7 MiB/s wr, 442 op/s 2024-05-08T20:24:44.471 INFO:tasks.workunit.client.1.smithi188.stdout: 10 90393 24.94 MB/sec execute 68 sec latency 112.827 ms 2024-05-08T20:24:44.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:44 smithi188 ceph-mon[48761]: pgmap v759: 129 pgs: 129 active+clean; 433 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 12 KiB/s rd, 9.7 MiB/s wr, 442 op/s 2024-05-08T20:24:44.746 INFO:tasks.workunit.client.0.smithi083.stdout: 10 156889 43.23 MB/sec execute 68 sec latency 116.899 ms 2024-05-08T20:24:45.471 INFO:tasks.workunit.client.1.smithi188.stdout: 10 90868 24.76 MB/sec execute 69 sec latency 87.972 ms 2024-05-08T20:24:45.747 INFO:tasks.workunit.client.0.smithi083.stdout: 10 157606 43.22 MB/sec execute 69 sec latency 125.464 ms 2024-05-08T20:24:46.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:46 smithi083 ceph-mon[38695]: pgmap v760: 129 pgs: 129 active+clean; 434 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 501 op/s 2024-05-08T20:24:46.471 INFO:tasks.workunit.client.1.smithi188.stdout: 10 91404 24.85 MB/sec execute 70 sec latency 75.541 ms 2024-05-08T20:24:46.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:46 smithi188 ceph-mon[48761]: pgmap v760: 129 pgs: 129 active+clean; 434 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 501 op/s 2024-05-08T20:24:46.746 INFO:tasks.workunit.client.0.smithi083.stdout: 10 158740 43.30 MB/sec execute 70 sec latency 273.024 ms 2024-05-08T20:24:47.471 INFO:tasks.workunit.client.1.smithi188.stdout: 10 91959 25.17 MB/sec execute 71 sec latency 69.087 ms 2024-05-08T20:24:47.748 INFO:tasks.workunit.client.0.smithi083.stdout: 10 160322 44.05 MB/sec execute 71 sec latency 125.464 ms 2024-05-08T20:24:48.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:48 smithi083 ceph-mon[38695]: pgmap v761: 129 pgs: 129 active+clean; 430 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 580 op/s 2024-05-08T20:24:48.471 INFO:tasks.workunit.client.1.smithi188.stdout: 10 92296 25.16 MB/sec execute 72 sec latency 110.648 ms 2024-05-08T20:24:48.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:48 smithi188 ceph-mon[48761]: pgmap v761: 129 pgs: 129 active+clean; 430 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 580 op/s 2024-05-08T20:24:48.748 INFO:tasks.workunit.client.0.smithi083.stdout: 10 160446 43.48 MB/sec execute 72 sec latency 125.376 ms 2024-05-08T20:24:49.472 INFO:tasks.workunit.client.1.smithi188.stdout: 10 92686 25.14 MB/sec execute 73 sec latency 38.981 ms 2024-05-08T20:24:49.750 INFO:tasks.workunit.client.0.smithi083.stdout: 10 161328 43.54 MB/sec execute 73 sec latency 136.215 ms 2024-05-08T20:24:50.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:50 smithi083 ceph-mon[38695]: pgmap v762: 129 pgs: 129 active+clean; 431 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 626 op/s 2024-05-08T20:24:50.471 INFO:tasks.workunit.client.1.smithi188.stdout: 10 93090 24.94 MB/sec execute 74 sec latency 73.333 ms 2024-05-08T20:24:50.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:50 smithi188 ceph-mon[48761]: pgmap v762: 129 pgs: 129 active+clean; 431 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 626 op/s 2024-05-08T20:24:50.753 INFO:tasks.workunit.client.0.smithi083.stdout: 10 162671 43.96 MB/sec execute 74 sec latency 145.392 ms 2024-05-08T20:24:51.472 INFO:tasks.workunit.client.1.smithi188.stdout: 10 93518 24.80 MB/sec execute 75 sec latency 72.549 ms 2024-05-08T20:24:51.753 INFO:tasks.workunit.client.0.smithi083.stdout: 10 163913 44.27 MB/sec execute 75 sec latency 122.766 ms 2024-05-08T20:24:52.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:52 smithi083 ceph-mon[38695]: pgmap v763: 129 pgs: 129 active+clean; 441 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 21 KiB/s rd, 13 MiB/s wr, 585 op/s 2024-05-08T20:24:52.472 INFO:tasks.workunit.client.1.smithi188.stdout: 10 93988 24.88 MB/sec execute 76 sec latency 102.652 ms 2024-05-08T20:24:52.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:52 smithi188 ceph-mon[48761]: pgmap v763: 129 pgs: 129 active+clean; 441 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 21 KiB/s rd, 13 MiB/s wr, 585 op/s 2024-05-08T20:24:52.753 INFO:tasks.workunit.client.0.smithi083.stdout: 10 164076 43.74 MB/sec execute 76 sec latency 133.524 ms 2024-05-08T20:24:53.472 INFO:tasks.workunit.client.1.smithi188.stdout: 10 94400 24.67 MB/sec execute 77 sec latency 130.487 ms 2024-05-08T20:24:53.753 INFO:tasks.workunit.client.0.smithi083.stdout: 10 165507 43.96 MB/sec execute 77 sec latency 128.357 ms 2024-05-08T20:24:54.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:54 smithi083 ceph-mon[38695]: pgmap v764: 129 pgs: 129 active+clean; 441 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 431 op/s 2024-05-08T20:24:54.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:24:54.472 INFO:tasks.workunit.client.1.smithi188.stdout: 10 94912 24.68 MB/sec execute 78 sec latency 85.393 ms 2024-05-08T20:24:54.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:54 smithi188 ceph-mon[48761]: pgmap v764: 129 pgs: 129 active+clean; 441 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 431 op/s 2024-05-08T20:24:54.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:24:54.755 INFO:tasks.workunit.client.0.smithi083.stdout: 10 166692 44.50 MB/sec execute 78 sec latency 111.932 ms 2024-05-08T20:24:55.472 INFO:tasks.workunit.client.1.smithi188.stdout: 10 95552 25.09 MB/sec execute 79 sec latency 75.169 ms 2024-05-08T20:24:55.753 INFO:tasks.workunit.client.0.smithi083.stdout: 10 167511 44.46 MB/sec execute 79 sec latency 109.905 ms 2024-05-08T20:24:56.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:56 smithi083 ceph-mon[38695]: pgmap v765: 129 pgs: 129 active+clean; 434 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 494 op/s 2024-05-08T20:24:56.473 INFO:tasks.workunit.client.1.smithi188.stdout: 10 95919 25.10 MB/sec execute 80 sec latency 70.136 ms 2024-05-08T20:24:56.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:56 smithi188 ceph-mon[48761]: pgmap v765: 129 pgs: 129 active+clean; 434 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 494 op/s 2024-05-08T20:24:56.753 INFO:tasks.workunit.client.0.smithi083.stdout: 10 168052 44.28 MB/sec execute 80 sec latency 112.816 ms 2024-05-08T20:24:57.473 INFO:tasks.workunit.client.1.smithi188.stdout: 10 96240 25.02 MB/sec execute 81 sec latency 78.351 ms 2024-05-08T20:24:57.754 INFO:tasks.workunit.client.0.smithi083.stdout: 10 168600 43.97 MB/sec execute 81 sec latency 497.534 ms 2024-05-08T20:24:58.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:24:58 smithi083 ceph-mon[38695]: pgmap v766: 129 pgs: 129 active+clean; 436 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 22 KiB/s rd, 11 MiB/s wr, 597 op/s 2024-05-08T20:24:58.473 INFO:tasks.workunit.client.1.smithi188.stdout: 10 96649 24.87 MB/sec execute 82 sec latency 74.734 ms 2024-05-08T20:24:58.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:24:58 smithi188 ceph-mon[48761]: pgmap v766: 129 pgs: 129 active+clean; 436 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 22 KiB/s rd, 11 MiB/s wr, 597 op/s 2024-05-08T20:24:58.753 INFO:tasks.workunit.client.0.smithi083.stdout: 10 168912 43.55 MB/sec execute 82 sec latency 461.401 ms 2024-05-08T20:24:59.473 INFO:tasks.workunit.client.1.smithi188.stdout: 10 96995 24.68 MB/sec execute 83 sec latency 103.732 ms 2024-05-08T20:24:59.754 INFO:tasks.workunit.client.0.smithi083.stdout: 10 169233 43.24 MB/sec execute 83 sec latency 463.551 ms 2024-05-08T20:25:00.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:00 smithi083 ceph-mon[38695]: pgmap v767: 129 pgs: 129 active+clean; 435 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 550 op/s 2024-05-08T20:25:00.473 INFO:tasks.workunit.client.1.smithi188.stdout: 10 97447 24.77 MB/sec execute 84 sec latency 111.775 ms 2024-05-08T20:25:00.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:00 smithi188 ceph-mon[48761]: pgmap v767: 129 pgs: 129 active+clean; 435 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 550 op/s 2024-05-08T20:25:00.754 INFO:tasks.workunit.client.0.smithi083.stdout: 10 169557 43.12 MB/sec execute 84 sec latency 465.051 ms 2024-05-08T20:25:01.473 INFO:tasks.workunit.client.1.smithi188.stdout: 10 97788 24.54 MB/sec execute 85 sec latency 88.985 ms 2024-05-08T20:25:01.754 INFO:tasks.workunit.client.0.smithi083.stdout: 10 169753 42.77 MB/sec execute 85 sec latency 494.199 ms 2024-05-08T20:25:02.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:02 smithi083 ceph-mon[38695]: pgmap v768: 129 pgs: 129 active+clean; 438 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 18 KiB/s rd, 13 MiB/s wr, 517 op/s 2024-05-08T20:25:02.474 INFO:tasks.workunit.client.1.smithi188.stdout: 10 98336 24.52 MB/sec execute 86 sec latency 67.868 ms 2024-05-08T20:25:02.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:02 smithi188 ceph-mon[48761]: pgmap v768: 129 pgs: 129 active+clean; 438 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 18 KiB/s rd, 13 MiB/s wr, 517 op/s 2024-05-08T20:25:02.754 INFO:tasks.workunit.client.0.smithi083.stdout: 10 170034 42.46 MB/sec execute 86 sec latency 531.578 ms 2024-05-08T20:25:03.474 INFO:tasks.workunit.client.1.smithi188.stdout: 10 98875 24.83 MB/sec execute 87 sec latency 73.342 ms 2024-05-08T20:25:03.754 INFO:tasks.workunit.client.0.smithi083.stdout: 10 171026 42.56 MB/sec execute 87 sec latency 106.981 ms 2024-05-08T20:25:04.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:04 smithi083 ceph-mon[38695]: pgmap v769: 129 pgs: 129 active+clean; 437 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 14 KiB/s rd, 9.1 MiB/s wr, 450 op/s 2024-05-08T20:25:04.474 INFO:tasks.workunit.client.1.smithi188.stdout: 10 99357 24.89 MB/sec execute 88 sec latency 91.924 ms 2024-05-08T20:25:04.598 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:04 smithi188 ceph-mon[48761]: pgmap v769: 129 pgs: 129 active+clean; 437 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 14 KiB/s rd, 9.1 MiB/s wr, 450 op/s 2024-05-08T20:25:04.754 INFO:tasks.workunit.client.0.smithi083.stdout: 10 171592 42.43 MB/sec execute 88 sec latency 97.016 ms 2024-05-08T20:25:05.474 INFO:tasks.workunit.client.1.smithi188.stdout: 10 99779 24.92 MB/sec execute 89 sec latency 96.687 ms 2024-05-08T20:25:05.754 INFO:tasks.workunit.client.0.smithi083.stdout: 10 173100 42.85 MB/sec execute 89 sec latency 130.756 ms 2024-05-08T20:25:06.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:06 smithi083 ceph-mon[38695]: pgmap v770: 129 pgs: 129 active+clean; 439 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 535 op/s 2024-05-08T20:25:06.474 INFO:tasks.workunit.client.1.smithi188.stdout: 10 100077 24.71 MB/sec execute 90 sec latency 105.140 ms 2024-05-08T20:25:06.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:06 smithi188 ceph-mon[48761]: pgmap v770: 129 pgs: 129 active+clean; 439 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 535 op/s 2024-05-08T20:25:06.754 INFO:tasks.workunit.client.0.smithi083.stdout: 10 174507 43.25 MB/sec execute 90 sec latency 138.372 ms 2024-05-08T20:25:07.474 INFO:tasks.workunit.client.1.smithi188.stdout: 10 100489 24.60 MB/sec execute 91 sec latency 36.866 ms 2024-05-08T20:25:07.754 INFO:tasks.workunit.client.0.smithi083.stdout: 10 174631 42.81 MB/sec execute 91 sec latency 112.077 ms 2024-05-08T20:25:08.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:08 smithi083 ceph-mon[38695]: pgmap v771: 129 pgs: 129 active+clean; 439 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 576 op/s 2024-05-08T20:25:08.474 INFO:tasks.workunit.client.1.smithi188.stdout: 10 100876 24.59 MB/sec execute 92 sec latency 131.403 ms 2024-05-08T20:25:08.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:08 smithi188 ceph-mon[48761]: pgmap v771: 129 pgs: 129 active+clean; 439 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 576 op/s 2024-05-08T20:25:08.756 INFO:tasks.workunit.client.0.smithi083.stdout: 10 175587 42.86 MB/sec execute 92 sec latency 221.387 ms 2024-05-08T20:25:09.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:25:09.474 INFO:tasks.workunit.client.1.smithi188.stdout: 10 101216 24.47 MB/sec execute 93 sec latency 117.424 ms 2024-05-08T20:25:09.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:25:09.756 INFO:tasks.workunit.client.0.smithi083.stdout: 10 176851 43.20 MB/sec execute 93 sec latency 122.492 ms 2024-05-08T20:25:10.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:10 smithi083 ceph-mon[38695]: pgmap v772: 129 pgs: 129 active+clean; 444 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 539 op/s 2024-05-08T20:25:10.475 INFO:tasks.workunit.client.1.smithi188.stdout: 10 101710 24.40 MB/sec execute 94 sec latency 80.006 ms 2024-05-08T20:25:10.598 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:10 smithi188 ceph-mon[48761]: pgmap v772: 129 pgs: 129 active+clean; 444 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 539 op/s 2024-05-08T20:25:10.756 INFO:tasks.workunit.client.0.smithi083.stdout: 10 178076 43.45 MB/sec execute 94 sec latency 107.450 ms 2024-05-08T20:25:11.477 INFO:tasks.workunit.client.1.smithi188.stdout: 10 102185 24.45 MB/sec execute 95 sec latency 82.757 ms 2024-05-08T20:25:11.756 INFO:tasks.workunit.client.0.smithi083.stdout: 10 178221 43.03 MB/sec execute 95 sec latency 113.896 ms 2024-05-08T20:25:12.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:12 smithi083 ceph-mon[38695]: pgmap v773: 129 pgs: 129 active+clean; 416 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 538 op/s 2024-05-08T20:25:12.475 INFO:tasks.workunit.client.1.smithi188.stdout: 10 102785 24.72 MB/sec execute 96 sec latency 48.896 ms 2024-05-08T20:25:12.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:12 smithi188 ceph-mon[48761]: pgmap v773: 129 pgs: 129 active+clean; 416 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 538 op/s 2024-05-08T20:25:12.757 INFO:tasks.workunit.client.0.smithi083.stdout: 10 179349 43.11 MB/sec execute 96 sec latency 137.324 ms 2024-05-08T20:25:13.475 INFO:tasks.workunit.client.1.smithi188.stdout: 10 103218 24.79 MB/sec execute 97 sec latency 84.510 ms 2024-05-08T20:25:13.757 INFO:tasks.workunit.client.0.smithi083.stdout: 10 180657 43.55 MB/sec execute 97 sec latency 122.271 ms 2024-05-08T20:25:14.476 INFO:tasks.workunit.client.1.smithi188.stdout: 10 103567 24.64 MB/sec execute 98 sec latency 87.472 ms 2024-05-08T20:25:14.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:14 smithi188 ceph-mon[48761]: pgmap v774: 129 pgs: 129 active+clean; 418 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 16 KiB/s rd, 8.8 MiB/s wr, 482 op/s 2024-05-08T20:25:14.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:14 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:25:14.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:14 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:25:14.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:14 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:25:14.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:14 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:25:14.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:14 smithi083 ceph-mon[38695]: pgmap v774: 129 pgs: 129 active+clean; 418 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 16 KiB/s rd, 8.8 MiB/s wr, 482 op/s 2024-05-08T20:25:14.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:14 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:25:14.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:14 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:25:14.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:14 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:25:14.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:14 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:25:14.757 INFO:tasks.workunit.client.0.smithi083.stdout: 10 181649 43.63 MB/sec execute 98 sec latency 128.694 ms 2024-05-08T20:25:15.476 INFO:tasks.workunit.client.1.smithi188.stdout: 10 103923 24.52 MB/sec execute 99 sec latency 105.477 ms 2024-05-08T20:25:15.760 INFO:tasks.workunit.client.0.smithi083.stdout: 10 181850 43.24 MB/sec execute 99 sec latency 101.218 ms 2024-05-08T20:25:16.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:16 smithi188 ceph-mon[48761]: pgmap v775: 129 pgs: 129 active+clean; 417 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 535 op/s 2024-05-08T20:25:16.477 INFO:tasks.workunit.client.1.smithi188.stdout: 10 104269 24.41 MB/sec execute 100 sec latency 117.668 ms 2024-05-08T20:25:16.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:16 smithi083 ceph-mon[38695]: pgmap v775: 129 pgs: 129 active+clean; 417 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 535 op/s 2024-05-08T20:25:16.759 INFO:tasks.workunit.client.0.smithi083.stdout: 10 183319 43.44 MB/sec execute 100 sec latency 134.989 ms 2024-05-08T20:25:17.477 INFO:tasks.workunit.client.1.smithi188.stdout: 10 104657 24.41 MB/sec execute 101 sec latency 131.192 ms 2024-05-08T20:25:17.759 INFO:tasks.workunit.client.0.smithi083.stdout: 10 184689 43.97 MB/sec execute 101 sec latency 139.048 ms 2024-05-08T20:25:18.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:18 smithi083 ceph-mon[38695]: pgmap v776: 129 pgs: 129 active+clean; 435 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 20 KiB/s rd, 11 MiB/s wr, 530 op/s 2024-05-08T20:25:18.477 INFO:tasks.workunit.client.1.smithi188.stdout: 10 105101 24.30 MB/sec execute 102 sec latency 127.169 ms 2024-05-08T20:25:18.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:18 smithi188 ceph-mon[48761]: pgmap v776: 129 pgs: 129 active+clean; 435 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 20 KiB/s rd, 11 MiB/s wr, 530 op/s 2024-05-08T20:25:18.759 INFO:tasks.workunit.client.0.smithi083.stdout: 10 185194 43.79 MB/sec execute 102 sec latency 317.688 ms 2024-05-08T20:25:19.477 INFO:tasks.workunit.client.1.smithi188.stdout: 10 105582 24.30 MB/sec execute 103 sec latency 70.947 ms 2024-05-08T20:25:19.760 INFO:tasks.workunit.client.0.smithi083.stdout: 10 185389 43.42 MB/sec execute 103 sec latency 122.144 ms 2024-05-08T20:25:20.477 INFO:tasks.workunit.client.1.smithi188.stdout: 10 106199 24.58 MB/sec execute 104 sec latency 55.291 ms 2024-05-08T20:25:20.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:20 smithi188 ceph-mon[48761]: pgmap v777: 129 pgs: 129 active+clean; 435 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 553 op/s 2024-05-08T20:25:20.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:20 smithi083 ceph-mon[38695]: pgmap v777: 129 pgs: 129 active+clean; 435 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 553 op/s 2024-05-08T20:25:20.762 INFO:tasks.workunit.client.0.smithi083.stdout: 10 186862 43.61 MB/sec execute 104 sec latency 117.499 ms 2024-05-08T20:25:21.477 INFO:tasks.workunit.client.1.smithi188.stdout: 10 106605 24.60 MB/sec execute 105 sec latency 79.605 ms 2024-05-08T20:25:21.762 INFO:tasks.workunit.client.0.smithi083.stdout: 10 187966 43.98 MB/sec execute 105 sec latency 140.869 ms 2024-05-08T20:25:22.477 INFO:tasks.workunit.client.1.smithi188.stdout: 10 107009 24.57 MB/sec execute 106 sec latency 65.501 ms 2024-05-08T20:25:22.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:22 smithi188 ceph-mon[48761]: pgmap v778: 129 pgs: 129 active+clean; 414 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 21 KiB/s rd, 11 MiB/s wr, 491 op/s 2024-05-08T20:25:22.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:22 smithi083 ceph-mon[38695]: pgmap v778: 129 pgs: 129 active+clean; 414 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 21 KiB/s rd, 11 MiB/s wr, 491 op/s 2024-05-08T20:25:22.762 INFO:tasks.workunit.client.0.smithi083.stdout: 10 188759 43.95 MB/sec execute 106 sec latency 117.346 ms 2024-05-08T20:25:23.477 INFO:tasks.workunit.client.1.smithi188.stdout: 10 107404 24.45 MB/sec execute 107 sec latency 123.679 ms 2024-05-08T20:25:23.762 INFO:tasks.workunit.client.0.smithi083.stdout: 10 189089 43.67 MB/sec execute 107 sec latency 108.363 ms 2024-05-08T20:25:24.477 INFO:tasks.workunit.client.1.smithi188.stdout: 10 107758 24.36 MB/sec execute 108 sec latency 117.165 ms 2024-05-08T20:25:24.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:24 smithi188 ceph-mon[48761]: pgmap v779: 129 pgs: 129 active+clean; 417 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 430 op/s 2024-05-08T20:25:24.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:24 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:25:24.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:24 smithi083 ceph-mon[38695]: pgmap v779: 129 pgs: 129 active+clean; 417 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 430 op/s 2024-05-08T20:25:24.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:25:24.762 INFO:tasks.workunit.client.0.smithi083.stdout: 10 190507 43.85 MB/sec execute 108 sec latency 134.171 ms 2024-05-08T20:25:25.477 INFO:tasks.workunit.client.1.smithi188.stdout: 10 108106 24.29 MB/sec execute 109 sec latency 144.285 ms 2024-05-08T20:25:25.762 INFO:tasks.workunit.client.0.smithi083.stdout: 10 192136 44.43 MB/sec execute 109 sec latency 115.909 ms 2024-05-08T20:25:26.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:26 smithi083 ceph-mon[38695]: pgmap v780: 129 pgs: 129 active+clean; 430 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 488 op/s 2024-05-08T20:25:26.477 INFO:tasks.workunit.client.1.smithi188.stdout: 10 108463 24.20 MB/sec execute 110 sec latency 131.565 ms 2024-05-08T20:25:26.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:26 smithi188 ceph-mon[48761]: pgmap v780: 129 pgs: 129 active+clean; 430 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 488 op/s 2024-05-08T20:25:26.762 INFO:tasks.workunit.client.0.smithi083.stdout: 10 192348 44.10 MB/sec execute 110 sec latency 119.725 ms 2024-05-08T20:25:27.477 INFO:tasks.workunit.client.1.smithi188.stdout: 10 108959 24.20 MB/sec execute 111 sec latency 88.091 ms 2024-05-08T20:25:27.762 INFO:tasks.workunit.client.0.smithi083.stdout: 10 193221 44.13 MB/sec execute 111 sec latency 127.176 ms 2024-05-08T20:25:28.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:28 smithi083 ceph-mon[38695]: pgmap v781: 129 pgs: 129 active+clean; 435 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 548 op/s 2024-05-08T20:25:28.477 INFO:tasks.workunit.client.1.smithi188.stdout: 10 109419 24.30 MB/sec execute 112 sec latency 52.262 ms 2024-05-08T20:25:28.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:28 smithi188 ceph-mon[48761]: pgmap v781: 129 pgs: 129 active+clean; 435 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 548 op/s 2024-05-08T20:25:28.762 INFO:tasks.workunit.client.0.smithi083.stdout: 10 194417 44.33 MB/sec execute 112 sec latency 135.227 ms 2024-05-08T20:25:29.477 INFO:tasks.workunit.client.1.smithi188.stdout: 10 109896 24.39 MB/sec execute 113 sec latency 70.152 ms 2024-05-08T20:25:29.762 INFO:tasks.workunit.client.0.smithi083.stdout: 10 194867 44.16 MB/sec execute 113 sec latency 328.410 ms 2024-05-08T20:25:30.477 INFO:tasks.workunit.client.1.smithi188.stdout: 10 110199 24.37 MB/sec execute 114 sec latency 55.433 ms 2024-05-08T20:25:30.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:30 smithi188 ceph-mon[48761]: pgmap v782: 129 pgs: 129 active+clean; 426 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 550 op/s 2024-05-08T20:25:30.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:30 smithi083 ceph-mon[38695]: pgmap v782: 129 pgs: 129 active+clean; 426 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 550 op/s 2024-05-08T20:25:30.762 INFO:tasks.workunit.client.0.smithi083.stdout: 10 195837 44.22 MB/sec execute 114 sec latency 342.075 ms 2024-05-08T20:25:31.477 INFO:tasks.workunit.client.1.smithi188.stdout: 10 110573 24.31 MB/sec execute 115 sec latency 93.546 ms 2024-05-08T20:25:31.762 INFO:tasks.workunit.client.0.smithi083.stdout: 10 196061 43.88 MB/sec execute 115 sec latency 119.223 ms 2024-05-08T20:25:32.479 INFO:tasks.workunit.client.1.smithi188.stdout: 10 110958 24.20 MB/sec execute 116 sec latency 120.046 ms 2024-05-08T20:25:32.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:32 smithi188 ceph-mon[48761]: pgmap v783: 129 pgs: 129 active+clean; 423 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 486 op/s 2024-05-08T20:25:32.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:32 smithi083 ceph-mon[38695]: pgmap v783: 129 pgs: 129 active+clean; 423 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 486 op/s 2024-05-08T20:25:32.762 INFO:tasks.workunit.client.0.smithi083.stdout: 10 197562 44.08 MB/sec execute 116 sec latency 152.913 ms 2024-05-08T20:25:33.479 INFO:tasks.workunit.client.1.smithi188.stdout: 10 111289 24.09 MB/sec execute 117 sec latency 100.895 ms 2024-05-08T20:25:33.763 INFO:tasks.workunit.client.0.smithi083.stdout: 10 198961 44.54 MB/sec execute 117 sec latency 143.122 ms 2024-05-08T20:25:34.479 INFO:tasks.workunit.client.1.smithi188.stdout: 10 111566 24.02 MB/sec execute 118 sec latency 135.523 ms 2024-05-08T20:25:34.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:34 smithi188 ceph-mon[48761]: pgmap v784: 129 pgs: 129 active+clean; 424 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 12 KiB/s rd, 9.7 MiB/s wr, 414 op/s 2024-05-08T20:25:34.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:34 smithi083 ceph-mon[38695]: pgmap v784: 129 pgs: 129 active+clean; 424 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 12 KiB/s rd, 9.7 MiB/s wr, 414 op/s 2024-05-08T20:25:34.762 INFO:tasks.workunit.client.0.smithi083.stdout: 10 199417 44.34 MB/sec execute 118 sec latency 133.876 ms 2024-05-08T20:25:35.479 INFO:tasks.workunit.client.1.smithi188.stdout: 10 112030 24.00 MB/sec execute 119 sec latency 31.621 ms 2024-05-08T20:25:35.763 INFO:tasks.workunit.client.0.smithi083.stdout: 10 199445 43.98 MB/sec execute 119 sec latency 530.098 ms 2024-05-08T20:25:36.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:36 smithi083 ceph-mon[38695]: pgmap v785: 129 pgs: 129 active+clean; 428 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 487 op/s 2024-05-08T20:25:36.479 INFO:tasks.workunit.client.1.smithi188.stdout: 10 112567 23.99 MB/sec execute 120 sec latency 62.917 ms 2024-05-08T20:25:36.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:36 smithi188 ceph-mon[48761]: pgmap v785: 129 pgs: 129 active+clean; 428 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 487 op/s 2024-05-08T20:25:36.762 INFO:tasks.workunit.client.0.smithi083.stdout: 10 199447 43.61 MB/sec execute 120 sec latency 721.615 ms 2024-05-08T20:25:37.479 INFO:tasks.workunit.client.1.smithi188.stdout: 10 113072 24.15 MB/sec execute 121 sec latency 63.700 ms 2024-05-08T20:25:37.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:37 smithi188 ceph-mon[48761]: pgmap v786: 129 pgs: 129 active+clean; 424 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 10 MiB/s wr, 531 op/s 2024-05-08T20:25:37.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:37 smithi083 ceph-mon[38695]: pgmap v786: 129 pgs: 129 active+clean; 424 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 10 MiB/s wr, 531 op/s 2024-05-08T20:25:37.763 INFO:tasks.workunit.client.0.smithi083.stdout: 10 199494 43.26 MB/sec execute 121 sec latency 554.822 ms 2024-05-08T20:25:38.479 INFO:tasks.workunit.client.1.smithi188.stdout: 10 113579 24.26 MB/sec execute 122 sec latency 67.455 ms 2024-05-08T20:25:38.763 INFO:tasks.workunit.client.0.smithi083.stdout: 10 199603 42.93 MB/sec execute 122 sec latency 477.068 ms 2024-05-08T20:25:39.479 INFO:tasks.workunit.client.1.smithi188.stdout: 10 113959 24.25 MB/sec execute 123 sec latency 87.242 ms 2024-05-08T20:25:39.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:39 smithi083 ceph-mon[38695]: pgmap v787: 129 pgs: 129 active+clean; 421 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 16 KiB/s rd, 9.3 MiB/s wr, 482 op/s 2024-05-08T20:25:39.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:25:39.763 INFO:tasks.workunit.client.0.smithi083.stdout: 10 201286 43.29 MB/sec execute 123 sec latency 111.673 ms 2024-05-08T20:25:39.846 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:39 smithi188 ceph-mon[48761]: pgmap v787: 129 pgs: 129 active+clean; 421 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 16 KiB/s rd, 9.3 MiB/s wr, 482 op/s 2024-05-08T20:25:39.846 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:25:40.479 INFO:tasks.workunit.client.1.smithi188.stdout: 10 114311 24.13 MB/sec execute 124 sec latency 82.181 ms 2024-05-08T20:25:40.763 INFO:tasks.workunit.client.0.smithi083.stdout: 10 202868 43.72 MB/sec execute 124 sec latency 127.998 ms 2024-05-08T20:25:41.482 INFO:tasks.workunit.client.1.smithi188.stdout: 10 114681 24.04 MB/sec execute 125 sec latency 148.310 ms 2024-05-08T20:25:41.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:41 smithi083 ceph-mon[38695]: pgmap v788: 129 pgs: 129 active+clean; 431 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 507 op/s 2024-05-08T20:25:41.763 INFO:tasks.workunit.client.0.smithi083.stdout: 10 202994 43.40 MB/sec execute 125 sec latency 115.095 ms 2024-05-08T20:25:41.845 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:41 smithi188 ceph-mon[48761]: pgmap v788: 129 pgs: 129 active+clean; 431 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 507 op/s 2024-05-08T20:25:42.481 INFO:tasks.workunit.client.1.smithi188.stdout: 10 115122 24.01 MB/sec execute 126 sec latency 108.774 ms 2024-05-08T20:25:42.763 INFO:tasks.workunit.client.0.smithi083.stdout: 10 204052 43.44 MB/sec execute 126 sec latency 120.839 ms 2024-05-08T20:25:43.481 INFO:tasks.workunit.client.1.smithi188.stdout: 10 115537 23.97 MB/sec execute 127 sec latency 108.571 ms 2024-05-08T20:25:43.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:43 smithi083 ceph-mon[38695]: pgmap v789: 129 pgs: 129 active+clean; 433 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 443 op/s 2024-05-08T20:25:43.763 INFO:tasks.workunit.client.0.smithi083.stdout: 10 205413 43.76 MB/sec execute 127 sec latency 116.253 ms 2024-05-08T20:25:43.846 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:43 smithi188 ceph-mon[48761]: pgmap v789: 129 pgs: 129 active+clean; 433 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 443 op/s 2024-05-08T20:25:44.481 INFO:tasks.workunit.client.1.smithi188.stdout: 10 116026 23.93 MB/sec execute 128 sec latency 85.351 ms 2024-05-08T20:25:44.763 INFO:tasks.workunit.client.0.smithi083.stdout: 10 206440 43.86 MB/sec execute 128 sec latency 200.694 ms 2024-05-08T20:25:45.481 INFO:tasks.workunit.client.1.smithi188.stdout: 10 116603 24.15 MB/sec execute 129 sec latency 27.253 ms 2024-05-08T20:25:45.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:45 smithi083 ceph-mon[38695]: pgmap v790: 129 pgs: 129 active+clean; 427 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 523 op/s 2024-05-08T20:25:45.763 INFO:tasks.workunit.client.0.smithi083.stdout: 10 206599 43.54 MB/sec execute 129 sec latency 104.022 ms 2024-05-08T20:25:45.845 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:45 smithi188 ceph-mon[48761]: pgmap v790: 129 pgs: 129 active+clean; 427 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 523 op/s 2024-05-08T20:25:46.481 INFO:tasks.workunit.client.1.smithi188.stdout: 10 117108 24.23 MB/sec execute 130 sec latency 69.320 ms 2024-05-08T20:25:46.763 INFO:tasks.workunit.client.0.smithi083.stdout: 10 207855 43.64 MB/sec execute 130 sec latency 121.501 ms 2024-05-08T20:25:47.483 INFO:tasks.workunit.client.1.smithi188.stdout: 10 117453 24.21 MB/sec execute 131 sec latency 76.160 ms 2024-05-08T20:25:47.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:47 smithi083 ceph-mon[38695]: pgmap v791: 129 pgs: 129 active+clean; 418 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 11 MiB/s wr, 579 op/s 2024-05-08T20:25:47.763 INFO:tasks.workunit.client.0.smithi083.stdout: 10 209046 43.93 MB/sec execute 131 sec latency 118.312 ms 2024-05-08T20:25:47.848 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:47 smithi188 ceph-mon[48761]: pgmap v791: 129 pgs: 129 active+clean; 418 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 11 MiB/s wr, 579 op/s 2024-05-08T20:25:48.483 INFO:tasks.workunit.client.1.smithi188.stdout: 10 117787 24.09 MB/sec execute 132 sec latency 132.190 ms 2024-05-08T20:25:48.763 INFO:tasks.workunit.client.0.smithi083.stdout: 10 210019 43.98 MB/sec execute 132 sec latency 125.534 ms 2024-05-08T20:25:49.483 INFO:tasks.workunit.client.1.smithi188.stdout: 10 118195 24.02 MB/sec execute 133 sec latency 83.924 ms 2024-05-08T20:25:49.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:49 smithi083 ceph-mon[38695]: pgmap v792: 129 pgs: 129 active+clean; 419 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 467 op/s 2024-05-08T20:25:49.763 INFO:tasks.workunit.client.0.smithi083.stdout: 10 210163 43.68 MB/sec execute 133 sec latency 304.846 ms 2024-05-08T20:25:49.850 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:49 smithi188 ceph-mon[48761]: pgmap v792: 129 pgs: 129 active+clean; 419 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 467 op/s 2024-05-08T20:25:50.483 INFO:tasks.workunit.client.1.smithi188.stdout: 10 118624 24.00 MB/sec execute 134 sec latency 83.433 ms 2024-05-08T20:25:50.763 INFO:tasks.workunit.client.0.smithi083.stdout: 10 211603 43.81 MB/sec execute 134 sec latency 122.360 ms 2024-05-08T20:25:51.483 INFO:tasks.workunit.client.1.smithi188.stdout: 10 119000 23.95 MB/sec execute 135 sec latency 139.148 ms 2024-05-08T20:25:51.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:51 smithi083 ceph-mon[38695]: pgmap v793: 129 pgs: 129 active+clean; 435 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 17 KiB/s rd, 13 MiB/s wr, 548 op/s 2024-05-08T20:25:51.763 INFO:tasks.workunit.client.0.smithi083.stdout: 10 212782 44.12 MB/sec execute 135 sec latency 125.970 ms 2024-05-08T20:25:51.847 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:51 smithi188 ceph-mon[48761]: pgmap v793: 129 pgs: 129 active+clean; 435 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 17 KiB/s rd, 13 MiB/s wr, 548 op/s 2024-05-08T20:25:52.483 INFO:tasks.workunit.client.1.smithi188.stdout: 10 119567 23.95 MB/sec execute 136 sec latency 78.474 ms 2024-05-08T20:25:52.763 INFO:tasks.workunit.client.0.smithi083.stdout: 10 213573 44.09 MB/sec execute 136 sec latency 117.111 ms 2024-05-08T20:25:53.483 INFO:tasks.workunit.client.1.smithi188.stdout: 10 120117 24.09 MB/sec execute 137 sec latency 80.430 ms 2024-05-08T20:25:53.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:53 smithi083 ceph-mon[38695]: pgmap v794: 129 pgs: 129 active+clean; 434 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 9.1 MiB/s wr, 443 op/s 2024-05-08T20:25:53.763 INFO:tasks.workunit.client.0.smithi083.stdout: 10 213910 43.87 MB/sec execute 137 sec latency 120.532 ms 2024-05-08T20:25:53.845 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:53 smithi188 ceph-mon[48761]: pgmap v794: 129 pgs: 129 active+clean; 434 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 9.1 MiB/s wr, 443 op/s 2024-05-08T20:25:54.483 INFO:tasks.workunit.client.1.smithi188.stdout: 10 120683 24.22 MB/sec execute 138 sec latency 63.903 ms 2024-05-08T20:25:54.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:25:54.763 INFO:tasks.workunit.client.0.smithi083.stdout: 10 215521 44.13 MB/sec execute 138 sec latency 127.019 ms 2024-05-08T20:25:54.846 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:25:55.483 INFO:tasks.workunit.client.1.smithi188.stdout: 10 121061 24.20 MB/sec execute 139 sec latency 83.887 ms 2024-05-08T20:25:55.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:55 smithi083 ceph-mon[38695]: pgmap v795: 129 pgs: 129 active+clean; 429 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 503 op/s 2024-05-08T20:25:55.763 INFO:tasks.workunit.client.0.smithi083.stdout: 10 217105 44.51 MB/sec execute 139 sec latency 151.270 ms 2024-05-08T20:25:55.847 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:55 smithi188 ceph-mon[48761]: pgmap v795: 129 pgs: 129 active+clean; 429 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 503 op/s 2024-05-08T20:25:56.483 INFO:tasks.workunit.client.1.smithi188.stdout: 10 121475 24.14 MB/sec execute 140 sec latency 39.263 ms 2024-05-08T20:25:56.764 INFO:tasks.workunit.client.0.smithi083.stdout: 10 217226 44.21 MB/sec execute 140 sec latency 125.098 ms 2024-05-08T20:25:57.483 INFO:tasks.workunit.client.1.smithi188.stdout: 10 121706 24.01 MB/sec execute 141 sec latency 109.354 ms 2024-05-08T20:25:57.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:57 smithi083 ceph-mon[38695]: pgmap v796: 129 pgs: 129 active+clean; 432 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 580 op/s 2024-05-08T20:25:57.763 INFO:tasks.workunit.client.0.smithi083.stdout: 10 218096 44.24 MB/sec execute 141 sec latency 106.261 ms 2024-05-08T20:25:57.845 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:57 smithi188 ceph-mon[48761]: pgmap v796: 129 pgs: 129 active+clean; 432 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 580 op/s 2024-05-08T20:25:58.487 INFO:tasks.workunit.client.1.smithi188.stdout: 10 122197 24.00 MB/sec execute 142 sec latency 32.907 ms 2024-05-08T20:25:58.764 INFO:tasks.workunit.client.0.smithi083.stdout: 10 219536 44.48 MB/sec execute 142 sec latency 123.078 ms 2024-05-08T20:25:59.487 INFO:tasks.workunit.client.1.smithi188.stdout: 10 122618 23.99 MB/sec execute 143 sec latency 99.917 ms 2024-05-08T20:25:59.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:25:59 smithi083 ceph-mon[38695]: pgmap v797: 129 pgs: 129 active+clean; 432 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 468 op/s 2024-05-08T20:25:59.763 INFO:tasks.workunit.client.0.smithi083.stdout: 10 220661 44.61 MB/sec execute 143 sec latency 162.552 ms 2024-05-08T20:25:59.845 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:25:59 smithi188 ceph-mon[48761]: pgmap v797: 129 pgs: 129 active+clean; 432 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 468 op/s 2024-05-08T20:26:00.487 INFO:tasks.workunit.client.1.smithi188.stdout: 10 123199 23.97 MB/sec execute 144 sec latency 67.312 ms 2024-05-08T20:26:00.764 INFO:tasks.workunit.client.0.smithi083.stdout: 10 220790 44.32 MB/sec execute 144 sec latency 124.616 ms 2024-05-08T20:26:01.487 INFO:tasks.workunit.client.1.smithi188.stdout: 10 123769 24.13 MB/sec execute 145 sec latency 93.700 ms 2024-05-08T20:26:01.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:01 smithi083 ceph-mon[38695]: pgmap v798: 129 pgs: 129 active+clean; 420 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 13 MiB/s wr, 631 op/s 2024-05-08T20:26:01.764 INFO:tasks.workunit.client.0.smithi083.stdout: 10 221869 44.35 MB/sec execute 145 sec latency 117.625 ms 2024-05-08T20:26:01.815 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:01 smithi188 ceph-mon[48761]: pgmap v798: 129 pgs: 129 active+clean; 420 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 13 MiB/s wr, 631 op/s 2024-05-08T20:26:02.487 INFO:tasks.workunit.client.1.smithi188.stdout: 10 124268 24.20 MB/sec execute 146 sec latency 62.854 ms 2024-05-08T20:26:02.764 INFO:tasks.workunit.client.0.smithi083.stdout: 10 223257 44.65 MB/sec execute 146 sec latency 113.954 ms 2024-05-08T20:26:03.487 INFO:tasks.workunit.client.1.smithi188.stdout: 10 124659 24.21 MB/sec execute 147 sec latency 100.919 ms 2024-05-08T20:26:03.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:03 smithi083 ceph-mon[38695]: pgmap v799: 129 pgs: 129 active+clean; 420 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 16 KiB/s rd, 9.1 MiB/s wr, 469 op/s 2024-05-08T20:26:03.764 INFO:tasks.workunit.client.0.smithi083.stdout: 10 224252 44.70 MB/sec execute 147 sec latency 114.302 ms 2024-05-08T20:26:03.847 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:03 smithi188 ceph-mon[48761]: pgmap v799: 129 pgs: 129 active+clean; 420 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 16 KiB/s rd, 9.1 MiB/s wr, 469 op/s 2024-05-08T20:26:04.489 INFO:tasks.workunit.client.1.smithi188.stdout: 10 124968 24.10 MB/sec execute 148 sec latency 113.343 ms 2024-05-08T20:26:04.764 INFO:tasks.workunit.client.0.smithi083.stdout: 10 224406 44.42 MB/sec execute 148 sec latency 260.811 ms 2024-05-08T20:26:05.489 INFO:tasks.workunit.client.1.smithi188.stdout: 10 125336 24.02 MB/sec execute 149 sec latency 84.665 ms 2024-05-08T20:26:05.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:05 smithi083 ceph-mon[38695]: pgmap v800: 129 pgs: 129 active+clean; 417 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 570 op/s 2024-05-08T20:26:05.764 INFO:tasks.workunit.client.0.smithi083.stdout: 10 225496 44.46 MB/sec execute 149 sec latency 382.132 ms 2024-05-08T20:26:05.846 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:05 smithi188 ceph-mon[48761]: pgmap v800: 129 pgs: 129 active+clean; 417 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 570 op/s 2024-05-08T20:26:06.489 INFO:tasks.workunit.client.1.smithi188.stdout: 10 125630 23.96 MB/sec execute 150 sec latency 111.458 ms 2024-05-08T20:26:06.764 INFO:tasks.workunit.client.0.smithi083.stdout: 10 226816 44.74 MB/sec execute 150 sec latency 126.637 ms 2024-05-08T20:26:07.489 INFO:tasks.workunit.client.1.smithi188.stdout: 10 126173 23.98 MB/sec execute 151 sec latency 27.384 ms 2024-05-08T20:26:07.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:07 smithi083 ceph-mon[38695]: pgmap v801: 129 pgs: 129 active+clean; 424 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 563 op/s 2024-05-08T20:26:07.764 INFO:tasks.workunit.client.0.smithi083.stdout: 10 227794 44.78 MB/sec execute 151 sec latency 127.870 ms 2024-05-08T20:26:07.847 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:07 smithi188 ceph-mon[48761]: pgmap v801: 129 pgs: 129 active+clean; 424 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 563 op/s 2024-05-08T20:26:08.489 INFO:tasks.workunit.client.1.smithi188.stdout: 10 126712 23.95 MB/sec execute 152 sec latency 62.132 ms 2024-05-08T20:26:08.764 INFO:tasks.workunit.client.0.smithi083.stdout: 10 227964 44.51 MB/sec execute 152 sec latency 121.976 ms 2024-05-08T20:26:09.195 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:26:09.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:26:09.489 INFO:tasks.workunit.client.1.smithi188.stdout: 10 127257 24.08 MB/sec execute 153 sec latency 85.236 ms 2024-05-08T20:26:09.764 INFO:tasks.workunit.client.0.smithi083.stdout: 10 229428 44.63 MB/sec execute 153 sec latency 142.809 ms 2024-05-08T20:26:10.195 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:09 smithi083 ceph-mon[38695]: pgmap v802: 129 pgs: 129 active+clean; 424 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 509 op/s 2024-05-08T20:26:10.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:09 smithi188 ceph-mon[48761]: pgmap v802: 129 pgs: 129 active+clean; 424 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 509 op/s 2024-05-08T20:26:10.489 INFO:tasks.workunit.client.1.smithi188.stdout: 10 127814 24.19 MB/sec execute 154 sec latency 83.042 ms 2024-05-08T20:26:10.765 INFO:tasks.workunit.client.0.smithi083.stdout: 10 230014 44.70 MB/sec execute 154 sec latency 522.766 ms 2024-05-08T20:26:11.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 128227 24.19 MB/sec execute 155 sec latency 82.242 ms 2024-05-08T20:26:11.764 INFO:tasks.workunit.client.0.smithi083.stdout: 10 230191 44.46 MB/sec execute 155 sec latency 547.618 ms 2024-05-08T20:26:12.195 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:11 smithi083 ceph-mon[38695]: pgmap v803: 129 pgs: 129 active+clean; 413 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 13 MiB/s wr, 630 op/s 2024-05-08T20:26:12.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:11 smithi188 ceph-mon[48761]: pgmap v803: 129 pgs: 129 active+clean; 413 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 13 MiB/s wr, 630 op/s 2024-05-08T20:26:12.489 INFO:tasks.workunit.client.1.smithi188.stdout: 10 128565 24.09 MB/sec execute 156 sec latency 101.753 ms 2024-05-08T20:26:12.764 INFO:tasks.workunit.client.0.smithi083.stdout: 10 230359 44.25 MB/sec execute 156 sec latency 560.923 ms 2024-05-08T20:26:13.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 128923 24.02 MB/sec execute 157 sec latency 95.565 ms 2024-05-08T20:26:13.764 INFO:tasks.workunit.client.0.smithi083.stdout: 10 230562 44.03 MB/sec execute 157 sec latency 580.921 ms 2024-05-08T20:26:14.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:13 smithi188 ceph-mon[48761]: pgmap v804: 129 pgs: 129 active+clean; 415 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 444 op/s 2024-05-08T20:26:14.449 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:13 smithi083 ceph-mon[38695]: pgmap v804: 129 pgs: 129 active+clean; 415 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 444 op/s 2024-05-08T20:26:14.489 INFO:tasks.workunit.client.1.smithi188.stdout: 10 129297 23.99 MB/sec execute 158 sec latency 114.503 ms 2024-05-08T20:26:14.764 INFO:tasks.workunit.client.0.smithi083.stdout: 10 231193 43.97 MB/sec execute 158 sec latency 494.282 ms 2024-05-08T20:26:15.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:14 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:26:15.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:14 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:26:15.348 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:14 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:26:15.348 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:14 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:26:15.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:14 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:26:15.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:14 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:26:15.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:14 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:26:15.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:14 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:26:15.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 129661 23.97 MB/sec execute 159 sec latency 137.059 ms 2024-05-08T20:26:15.764 INFO:tasks.workunit.client.0.smithi083.stdout: 10 231287 43.72 MB/sec execute 159 sec latency 505.585 ms 2024-05-08T20:26:16.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:15 smithi188 ceph-mon[48761]: pgmap v805: 129 pgs: 129 active+clean; 424 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 520 op/s 2024-05-08T20:26:16.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:15 smithi083 ceph-mon[38695]: pgmap v805: 129 pgs: 129 active+clean; 424 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 520 op/s 2024-05-08T20:26:16.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 130231 23.97 MB/sec execute 160 sec latency 74.762 ms 2024-05-08T20:26:16.764 INFO:tasks.workunit.client.0.smithi083.stdout: 10 231406 43.47 MB/sec execute 160 sec latency 387.085 ms 2024-05-08T20:26:17.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 130740 24.04 MB/sec execute 161 sec latency 81.393 ms 2024-05-08T20:26:17.764 INFO:tasks.workunit.client.0.smithi083.stdout: 10 232278 43.49 MB/sec execute 161 sec latency 106.596 ms 2024-05-08T20:26:18.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:17 smithi188 ceph-mon[48761]: pgmap v806: 129 pgs: 129 active+clean; 417 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 20 KiB/s rd, 12 MiB/s wr, 534 op/s 2024-05-08T20:26:18.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:17 smithi083 ceph-mon[38695]: pgmap v806: 129 pgs: 129 active+clean; 417 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 20 KiB/s rd, 12 MiB/s wr, 534 op/s 2024-05-08T20:26:18.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 131278 24.12 MB/sec execute 162 sec latency 62.740 ms 2024-05-08T20:26:18.764 INFO:tasks.workunit.client.0.smithi083.stdout: 10 233627 43.68 MB/sec execute 162 sec latency 135.364 ms 2024-05-08T20:26:19.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 131693 24.16 MB/sec execute 163 sec latency 68.192 ms 2024-05-08T20:26:19.765 INFO:tasks.workunit.client.0.smithi083.stdout: 10 234862 43.83 MB/sec execute 163 sec latency 116.515 ms 2024-05-08T20:26:20.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:19 smithi188 ceph-mon[48761]: pgmap v807: 129 pgs: 129 active+clean; 420 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 10 MiB/s wr, 534 op/s 2024-05-08T20:26:20.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:19 smithi083 ceph-mon[38695]: pgmap v807: 129 pgs: 129 active+clean; 420 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 10 MiB/s wr, 534 op/s 2024-05-08T20:26:20.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 132029 24.08 MB/sec execute 164 sec latency 108.024 ms 2024-05-08T20:26:20.765 INFO:tasks.workunit.client.0.smithi083.stdout: 10 235032 43.59 MB/sec execute 164 sec latency 115.558 ms 2024-05-08T20:26:21.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 132407 24.00 MB/sec execute 165 sec latency 81.109 ms 2024-05-08T20:26:21.766 INFO:tasks.workunit.client.0.smithi083.stdout: 10 236013 43.60 MB/sec execute 165 sec latency 265.047 ms 2024-05-08T20:26:22.195 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:22 smithi083 ceph-mon[38695]: pgmap v808: 129 pgs: 129 active+clean; 427 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 20 KiB/s rd, 13 MiB/s wr, 555 op/s 2024-05-08T20:26:22.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:22 smithi188 ceph-mon[48761]: pgmap v808: 129 pgs: 129 active+clean; 427 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 20 KiB/s rd, 13 MiB/s wr, 555 op/s 2024-05-08T20:26:22.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 132755 23.93 MB/sec execute 166 sec latency 97.181 ms 2024-05-08T20:26:22.766 INFO:tasks.workunit.client.0.smithi083.stdout: 10 237403 43.86 MB/sec execute 166 sec latency 110.754 ms 2024-05-08T20:26:23.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 133173 23.97 MB/sec execute 167 sec latency 105.771 ms 2024-05-08T20:26:23.767 INFO:tasks.workunit.client.0.smithi083.stdout: 10 238435 43.93 MB/sec execute 167 sec latency 116.420 ms 2024-05-08T20:26:24.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:24 smithi188 ceph-mon[48761]: pgmap v809: 129 pgs: 129 active+clean; 422 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 13 KiB/s rd, 9.7 MiB/s wr, 412 op/s 2024-05-08T20:26:24.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:24 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:26:24.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:24 smithi083 ceph-mon[38695]: pgmap v809: 129 pgs: 129 active+clean; 422 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 13 KiB/s rd, 9.7 MiB/s wr, 412 op/s 2024-05-08T20:26:24.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:26:24.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 133655 23.92 MB/sec execute 168 sec latency 105.788 ms 2024-05-08T20:26:24.767 INFO:tasks.workunit.client.0.smithi083.stdout: 10 238634 43.69 MB/sec execute 168 sec latency 117.679 ms 2024-05-08T20:26:25.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 134145 23.93 MB/sec execute 169 sec latency 84.372 ms 2024-05-08T20:26:25.767 INFO:tasks.workunit.client.0.smithi083.stdout: 10 240197 43.86 MB/sec execute 169 sec latency 122.741 ms 2024-05-08T20:26:26.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:26 smithi188 ceph-mon[48761]: pgmap v810: 129 pgs: 129 active+clean; 417 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 514 op/s 2024-05-08T20:26:26.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:26 smithi083 ceph-mon[38695]: pgmap v810: 129 pgs: 129 active+clean; 417 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 514 op/s 2024-05-08T20:26:26.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 134786 24.10 MB/sec execute 170 sec latency 74.757 ms 2024-05-08T20:26:26.767 INFO:tasks.workunit.client.0.smithi083.stdout: 10 241828 44.24 MB/sec execute 170 sec latency 124.971 ms 2024-05-08T20:26:27.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 135127 24.10 MB/sec execute 171 sec latency 94.628 ms 2024-05-08T20:26:27.767 INFO:tasks.workunit.client.0.smithi083.stdout: 10 242040 44.02 MB/sec execute 171 sec latency 113.962 ms 2024-05-08T20:26:28.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:28 smithi188 ceph-mon[48761]: pgmap v811: 129 pgs: 129 active+clean; 412 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 11 MiB/s wr, 552 op/s 2024-05-08T20:26:28.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:28 smithi083 ceph-mon[38695]: pgmap v811: 129 pgs: 129 active+clean; 412 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 11 MiB/s wr, 552 op/s 2024-05-08T20:26:28.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 135446 24.06 MB/sec execute 172 sec latency 101.904 ms 2024-05-08T20:26:28.767 INFO:tasks.workunit.client.0.smithi083.stdout: 10 242817 44.04 MB/sec execute 172 sec latency 129.246 ms 2024-05-08T20:26:29.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 135784 23.98 MB/sec execute 173 sec latency 109.249 ms 2024-05-08T20:26:29.767 INFO:tasks.workunit.client.0.smithi083.stdout: 10 244197 44.21 MB/sec execute 173 sec latency 134.777 ms 2024-05-08T20:26:30.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:30 smithi188 ceph-mon[48761]: pgmap v812: 129 pgs: 129 active+clean; 416 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 494 op/s 2024-05-08T20:26:30.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:30 smithi083 ceph-mon[38695]: pgmap v812: 129 pgs: 129 active+clean; 416 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 494 op/s 2024-05-08T20:26:30.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 136109 23.89 MB/sec execute 174 sec latency 117.635 ms 2024-05-08T20:26:30.767 INFO:tasks.workunit.client.0.smithi083.stdout: 10 245482 44.35 MB/sec execute 174 sec latency 130.700 ms 2024-05-08T20:26:31.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 136545 23.92 MB/sec execute 175 sec latency 117.368 ms 2024-05-08T20:26:31.767 INFO:tasks.workunit.client.0.smithi083.stdout: 10 245611 44.11 MB/sec execute 175 sec latency 117.187 ms 2024-05-08T20:26:32.325 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:32 smithi188 ceph-mon[48761]: pgmap v813: 129 pgs: 129 active+clean; 422 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 520 op/s 2024-05-08T20:26:32.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:32 smithi083 ceph-mon[38695]: pgmap v813: 129 pgs: 129 active+clean; 422 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 520 op/s 2024-05-08T20:26:32.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 136932 23.87 MB/sec execute 176 sec latency 134.706 ms 2024-05-08T20:26:32.767 INFO:tasks.workunit.client.0.smithi083.stdout: 10 246652 44.14 MB/sec execute 176 sec latency 127.721 ms 2024-05-08T20:26:33.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 137464 23.87 MB/sec execute 177 sec latency 84.849 ms 2024-05-08T20:26:33.767 INFO:tasks.workunit.client.0.smithi083.stdout: 10 248013 44.36 MB/sec execute 177 sec latency 129.352 ms 2024-05-08T20:26:34.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:34 smithi188 ceph-mon[48761]: pgmap v814: 129 pgs: 129 active+clean; 423 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 8.8 MiB/s wr, 431 op/s 2024-05-08T20:26:34.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:34 smithi083 ceph-mon[38695]: pgmap v814: 129 pgs: 129 active+clean; 423 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 8.8 MiB/s wr, 431 op/s 2024-05-08T20:26:34.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 137917 23.93 MB/sec execute 178 sec latency 98.492 ms 2024-05-08T20:26:34.767 INFO:tasks.workunit.client.0.smithi083.stdout: 10 249066 44.43 MB/sec execute 178 sec latency 119.883 ms 2024-05-08T20:26:35.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 138413 23.97 MB/sec execute 179 sec latency 77.561 ms 2024-05-08T20:26:35.767 INFO:tasks.workunit.client.0.smithi083.stdout: 10 249287 44.21 MB/sec execute 179 sec latency 109.984 ms 2024-05-08T20:26:36.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:36 smithi188 ceph-mon[48761]: pgmap v815: 129 pgs: 129 active+clean; 423 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 567 op/s 2024-05-08T20:26:36.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:36 smithi083 ceph-mon[38695]: pgmap v815: 129 pgs: 129 active+clean; 423 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 567 op/s 2024-05-08T20:26:36.490 INFO:tasks.workunit.client.1.smithi188.stdout: 10 138845 24.03 MB/sec execute 180 sec latency 84.593 ms 2024-05-08T20:26:36.767 INFO:tasks.workunit.client.0.smithi083.stdout: 10 250735 44.32 MB/sec execute 180 sec latency 136.478 ms 2024-05-08T20:26:37.492 INFO:tasks.workunit.client.1.smithi188.stdout: 10 139173 23.95 MB/sec execute 181 sec latency 86.538 ms 2024-05-08T20:26:37.767 INFO:tasks.workunit.client.0.smithi083.stdout: 10 251464 44.41 MB/sec execute 181 sec latency 358.865 ms 2024-05-08T20:26:38.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:38 smithi188 ceph-mon[48761]: pgmap v816: 129 pgs: 129 active+clean; 424 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 11 MiB/s wr, 510 op/s 2024-05-08T20:26:38.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:38 smithi083 ceph-mon[38695]: pgmap v816: 129 pgs: 129 active+clean; 424 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 11 MiB/s wr, 510 op/s 2024-05-08T20:26:38.492 INFO:tasks.workunit.client.1.smithi188.stdout: 10 139523 23.86 MB/sec execute 182 sec latency 129.088 ms 2024-05-08T20:26:38.767 INFO:tasks.workunit.client.0.smithi083.stdout: 10 252615 44.51 MB/sec execute 182 sec latency 134.933 ms 2024-05-08T20:26:39.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:26:39.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:26:39.492 INFO:tasks.workunit.client.1.smithi188.stdout: 10 139922 23.83 MB/sec execute 183 sec latency 86.689 ms 2024-05-08T20:26:39.767 INFO:tasks.workunit.client.0.smithi083.stdout: 10 252813 44.29 MB/sec execute 183 sec latency 110.829 ms 2024-05-08T20:26:40.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:40 smithi188 ceph-mon[48761]: pgmap v817: 129 pgs: 129 active+clean; 435 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 509 op/s 2024-05-08T20:26:40.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:40 smithi083 ceph-mon[38695]: pgmap v817: 129 pgs: 129 active+clean; 435 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 509 op/s 2024-05-08T20:26:40.492 INFO:tasks.workunit.client.1.smithi188.stdout: 10 140318 23.85 MB/sec execute 184 sec latency 124.901 ms 2024-05-08T20:26:40.767 INFO:tasks.workunit.client.0.smithi083.stdout: 10 254252 44.39 MB/sec execute 184 sec latency 112.746 ms 2024-05-08T20:26:41.493 INFO:tasks.workunit.client.1.smithi188.stdout: 10 140883 23.83 MB/sec execute 185 sec latency 88.087 ms 2024-05-08T20:26:41.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 255628 44.68 MB/sec execute 185 sec latency 129.801 ms 2024-05-08T20:26:42.345 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:42 smithi188 ceph-mon[48761]: pgmap v818: 129 pgs: 129 active+clean; 433 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 568 op/s 2024-05-08T20:26:42.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:42 smithi083 ceph-mon[38695]: pgmap v818: 129 pgs: 129 active+clean; 433 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 568 op/s 2024-05-08T20:26:42.493 INFO:tasks.workunit.client.1.smithi188.stdout: 10 141322 23.86 MB/sec execute 186 sec latency 89.229 ms 2024-05-08T20:26:42.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 256161 44.58 MB/sec execute 186 sec latency 184.200 ms 2024-05-08T20:26:43.493 INFO:tasks.workunit.client.1.smithi188.stdout: 10 141922 23.96 MB/sec execute 187 sec latency 80.036 ms 2024-05-08T20:26:43.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 256378 44.37 MB/sec execute 187 sec latency 133.081 ms 2024-05-08T20:26:44.351 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:44 smithi188 ceph-mon[48761]: pgmap v819: 129 pgs: 129 active+clean; 431 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 9.3 MiB/s wr, 496 op/s 2024-05-08T20:26:44.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:44 smithi083 ceph-mon[38695]: pgmap v819: 129 pgs: 129 active+clean; 431 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 9.3 MiB/s wr, 496 op/s 2024-05-08T20:26:44.493 INFO:tasks.workunit.client.1.smithi188.stdout: 10 142327 24.00 MB/sec execute 188 sec latency 60.728 ms 2024-05-08T20:26:44.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 257923 44.52 MB/sec execute 188 sec latency 133.268 ms 2024-05-08T20:26:45.493 INFO:tasks.workunit.client.1.smithi188.stdout: 10 142665 23.93 MB/sec execute 189 sec latency 122.271 ms 2024-05-08T20:26:45.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 259558 44.85 MB/sec execute 189 sec latency 136.847 ms 2024-05-08T20:26:46.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:46 smithi188 ceph-mon[48761]: pgmap v820: 129 pgs: 129 active+clean; 422 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 504 op/s 2024-05-08T20:26:46.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:46 smithi083 ceph-mon[38695]: pgmap v820: 129 pgs: 129 active+clean; 422 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 504 op/s 2024-05-08T20:26:46.493 INFO:tasks.workunit.client.1.smithi188.stdout: 10 143020 23.86 MB/sec execute 190 sec latency 110.180 ms 2024-05-08T20:26:46.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 259766 44.65 MB/sec execute 190 sec latency 107.563 ms 2024-05-08T20:26:47.493 INFO:tasks.workunit.client.1.smithi188.stdout: 10 143407 23.83 MB/sec execute 191 sec latency 129.231 ms 2024-05-08T20:26:47.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 259934 44.44 MB/sec execute 191 sec latency 276.948 ms 2024-05-08T20:26:48.352 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:48 smithi188 ceph-mon[48761]: pgmap v821: 129 pgs: 129 active+clean; 430 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 20 KiB/s rd, 12 MiB/s wr, 493 op/s 2024-05-08T20:26:48.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:48 smithi083 ceph-mon[38695]: pgmap v821: 129 pgs: 129 active+clean; 430 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 20 KiB/s rd, 12 MiB/s wr, 493 op/s 2024-05-08T20:26:48.493 INFO:tasks.workunit.client.1.smithi188.stdout: 10 143765 23.83 MB/sec execute 192 sec latency 176.243 ms 2024-05-08T20:26:48.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 260239 44.32 MB/sec execute 192 sec latency 530.506 ms 2024-05-08T20:26:49.493 INFO:tasks.workunit.client.1.smithi188.stdout: 10 144235 23.81 MB/sec execute 193 sec latency 90.388 ms 2024-05-08T20:26:49.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 260643 44.21 MB/sec execute 193 sec latency 543.727 ms 2024-05-08T20:26:50.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:50 smithi188 ceph-mon[48761]: pgmap v822: 129 pgs: 129 active+clean; 434 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 524 op/s 2024-05-08T20:26:50.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:50 smithi083 ceph-mon[38695]: pgmap v822: 129 pgs: 129 active+clean; 434 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 524 op/s 2024-05-08T20:26:50.493 INFO:tasks.workunit.client.1.smithi188.stdout: 10 144687 23.78 MB/sec execute 194 sec latency 89.429 ms 2024-05-08T20:26:50.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 260860 43.99 MB/sec execute 194 sec latency 472.032 ms 2024-05-08T20:26:51.493 INFO:tasks.workunit.client.1.smithi188.stdout: 10 145241 23.89 MB/sec execute 195 sec latency 65.043 ms 2024-05-08T20:26:51.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 261142 43.81 MB/sec execute 195 sec latency 486.966 ms 2024-05-08T20:26:52.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:52 smithi188 ceph-mon[48761]: pgmap v823: 129 pgs: 129 active+clean; 418 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 11 MiB/s wr, 522 op/s 2024-05-08T20:26:52.406 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:52 smithi083 ceph-mon[38695]: pgmap v823: 129 pgs: 129 active+clean; 418 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 11 MiB/s wr, 522 op/s 2024-05-08T20:26:52.493 INFO:tasks.workunit.client.1.smithi188.stdout: 10 145702 23.94 MB/sec execute 196 sec latency 75.895 ms 2024-05-08T20:26:52.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 261480 43.68 MB/sec execute 196 sec latency 458.902 ms 2024-05-08T20:26:53.493 INFO:tasks.workunit.client.1.smithi188.stdout: 10 146051 23.91 MB/sec execute 197 sec latency 107.866 ms 2024-05-08T20:26:53.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 263192 44.02 MB/sec execute 197 sec latency 131.859 ms 2024-05-08T20:26:54.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:54 smithi188 ceph-mon[48761]: pgmap v824: 129 pgs: 129 active+clean; 414 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 8.5 MiB/s wr, 407 op/s 2024-05-08T20:26:54.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:26:54.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:54 smithi083 ceph-mon[38695]: pgmap v824: 129 pgs: 129 active+clean; 414 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 8.5 MiB/s wr, 407 op/s 2024-05-08T20:26:54.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:26:54.493 INFO:tasks.workunit.client.1.smithi188.stdout: 10 146392 23.85 MB/sec execute 198 sec latency 114.243 ms 2024-05-08T20:26:54.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 263317 43.82 MB/sec execute 198 sec latency 105.899 ms 2024-05-08T20:26:55.495 INFO:tasks.workunit.client.1.smithi188.stdout: 10 146816 23.79 MB/sec execute 199 sec latency 82.388 ms 2024-05-08T20:26:55.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 264277 43.84 MB/sec execute 199 sec latency 120.803 ms 2024-05-08T20:26:56.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:56 smithi083 ceph-mon[38695]: pgmap v825: 129 pgs: 129 active+clean; 425 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 11 MiB/s wr, 496 op/s 2024-05-08T20:26:56.493 INFO:tasks.workunit.client.1.smithi188.stdout: 10 147247 23.82 MB/sec execute 200 sec latency 107.974 ms 2024-05-08T20:26:56.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:56 smithi188 ceph-mon[48761]: pgmap v825: 129 pgs: 129 active+clean; 425 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 11 MiB/s wr, 496 op/s 2024-05-08T20:26:56.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 265625 44.01 MB/sec execute 200 sec latency 120.154 ms 2024-05-08T20:26:57.493 INFO:tasks.workunit.client.1.smithi188.stdout: 10 147606 23.79 MB/sec execute 201 sec latency 146.122 ms 2024-05-08T20:26:57.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 266790 44.10 MB/sec execute 201 sec latency 161.404 ms 2024-05-08T20:26:58.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:26:58 smithi083 ceph-mon[38695]: pgmap v826: 129 pgs: 129 active+clean; 436 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 20 KiB/s rd, 11 MiB/s wr, 547 op/s 2024-05-08T20:26:58.494 INFO:tasks.workunit.client.1.smithi188.stdout: 10 148114 23.75 MB/sec execute 202 sec latency 95.258 ms 2024-05-08T20:26:58.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:26:58 smithi188 ceph-mon[48761]: pgmap v826: 129 pgs: 129 active+clean; 436 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 20 KiB/s rd, 11 MiB/s wr, 547 op/s 2024-05-08T20:26:58.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 266945 43.90 MB/sec execute 202 sec latency 154.591 ms 2024-05-08T20:26:59.494 INFO:tasks.workunit.client.1.smithi188.stdout: 10 148534 23.78 MB/sec execute 203 sec latency 92.516 ms 2024-05-08T20:26:59.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 268043 43.94 MB/sec execute 203 sec latency 134.122 ms 2024-05-08T20:27:00.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:00 smithi083 ceph-mon[38695]: pgmap v827: 129 pgs: 129 active+clean; 432 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 9.5 MiB/s wr, 483 op/s 2024-05-08T20:27:00.494 INFO:tasks.workunit.client.1.smithi188.stdout: 10 149074 23.85 MB/sec execute 204 sec latency 64.282 ms 2024-05-08T20:27:00.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:00 smithi188 ceph-mon[48761]: pgmap v827: 129 pgs: 129 active+clean; 432 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 9.5 MiB/s wr, 483 op/s 2024-05-08T20:27:00.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 269316 44.12 MB/sec execute 204 sec latency 136.845 ms 2024-05-08T20:27:01.495 INFO:tasks.workunit.client.1.smithi188.stdout: 10 149426 23.87 MB/sec execute 205 sec latency 83.457 ms 2024-05-08T20:27:01.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 270341 44.18 MB/sec execute 205 sec latency 130.450 ms 2024-05-08T20:27:02.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:02 smithi083 ceph-mon[38695]: pgmap v828: 129 pgs: 129 active+clean; 433 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 554 op/s 2024-05-08T20:27:02.495 INFO:tasks.workunit.client.1.smithi188.stdout: 10 149752 23.83 MB/sec execute 206 sec latency 74.655 ms 2024-05-08T20:27:02.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:02 smithi188 ceph-mon[48761]: pgmap v828: 129 pgs: 129 active+clean; 433 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 554 op/s 2024-05-08T20:27:02.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 270536 43.99 MB/sec execute 206 sec latency 125.411 ms 2024-05-08T20:27:03.495 INFO:tasks.workunit.client.1.smithi188.stdout: 10 150094 23.75 MB/sec execute 207 sec latency 82.084 ms 2024-05-08T20:27:03.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 271975 44.07 MB/sec execute 207 sec latency 127.224 ms 2024-05-08T20:27:04.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:04 smithi083 ceph-mon[38695]: pgmap v829: 129 pgs: 129 active+clean; 433 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 13 KiB/s rd, 9.7 MiB/s wr, 443 op/s 2024-05-08T20:27:04.495 INFO:tasks.workunit.client.1.smithi188.stdout: 10 150423 23.69 MB/sec execute 208 sec latency 83.623 ms 2024-05-08T20:27:04.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:04 smithi188 ceph-mon[48761]: pgmap v829: 129 pgs: 129 active+clean; 433 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 13 KiB/s rd, 9.7 MiB/s wr, 443 op/s 2024-05-08T20:27:04.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 273112 44.26 MB/sec execute 208 sec latency 111.887 ms 2024-05-08T20:27:05.495 INFO:tasks.workunit.client.1.smithi188.stdout: 10 150808 23.70 MB/sec execute 209 sec latency 120.091 ms 2024-05-08T20:27:05.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 273924 44.25 MB/sec execute 209 sec latency 111.733 ms 2024-05-08T20:27:06.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:06 smithi083 ceph-mon[38695]: pgmap v830: 129 pgs: 129 active+clean; 442 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 562 op/s 2024-05-08T20:27:06.495 INFO:tasks.workunit.client.1.smithi188.stdout: 10 151245 23.71 MB/sec execute 210 sec latency 129.444 ms 2024-05-08T20:27:06.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:06 smithi188 ceph-mon[48761]: pgmap v830: 129 pgs: 129 active+clean; 442 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 562 op/s 2024-05-08T20:27:06.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 274421 44.16 MB/sec execute 210 sec latency 141.274 ms 2024-05-08T20:27:07.495 INFO:tasks.workunit.client.1.smithi188.stdout: 10 151728 23.67 MB/sec execute 211 sec latency 67.648 ms 2024-05-08T20:27:07.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 275976 44.35 MB/sec execute 211 sec latency 125.711 ms 2024-05-08T20:27:08.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:08 smithi188 ceph-mon[48761]: pgmap v831: 129 pgs: 129 active+clean; 443 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 540 op/s 2024-05-08T20:27:08.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:08 smithi083 ceph-mon[38695]: pgmap v831: 129 pgs: 129 active+clean; 443 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 540 op/s 2024-05-08T20:27:08.495 INFO:tasks.workunit.client.1.smithi188.stdout: 10 152210 23.72 MB/sec execute 212 sec latency 103.921 ms 2024-05-08T20:27:08.768 INFO:tasks.workunit.client.0.smithi083.stdout: 10 277371 44.52 MB/sec execute 212 sec latency 247.443 ms 2024-05-08T20:27:09.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:27:09.495 INFO:tasks.workunit.client.1.smithi188.stdout: 10 152736 23.77 MB/sec execute 213 sec latency 79.029 ms 2024-05-08T20:27:09.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:27:09.769 INFO:tasks.workunit.client.0.smithi083.stdout: 10 277499 44.32 MB/sec execute 213 sec latency 112.685 ms 2024-05-08T20:27:10.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:10 smithi083 ceph-mon[38695]: pgmap v832: 129 pgs: 129 active+clean; 434 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 13 KiB/s rd, 9.6 MiB/s wr, 476 op/s 2024-05-08T20:27:10.495 INFO:tasks.workunit.client.1.smithi188.stdout: 10 153164 23.81 MB/sec execute 214 sec latency 95.709 ms 2024-05-08T20:27:10.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:10 smithi188 ceph-mon[48761]: pgmap v832: 129 pgs: 129 active+clean; 434 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 13 KiB/s rd, 9.6 MiB/s wr, 476 op/s 2024-05-08T20:27:10.769 INFO:tasks.workunit.client.0.smithi083.stdout: 10 278367 44.33 MB/sec execute 214 sec latency 138.141 ms 2024-05-08T20:27:11.495 INFO:tasks.workunit.client.1.smithi188.stdout: 10 153521 23.75 MB/sec execute 215 sec latency 133.597 ms 2024-05-08T20:27:11.769 INFO:tasks.workunit.client.0.smithi083.stdout: 10 279631 44.46 MB/sec execute 215 sec latency 138.724 ms 2024-05-08T20:27:12.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:12 smithi083 ceph-mon[38695]: pgmap v833: 129 pgs: 129 active+clean; 440 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 566 op/s 2024-05-08T20:27:12.495 INFO:tasks.workunit.client.1.smithi188.stdout: 10 153907 23.69 MB/sec execute 216 sec latency 98.167 ms 2024-05-08T20:27:12.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:12 smithi188 ceph-mon[48761]: pgmap v833: 129 pgs: 129 active+clean; 440 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 566 op/s 2024-05-08T20:27:12.769 INFO:tasks.workunit.client.0.smithi083.stdout: 10 280925 44.58 MB/sec execute 216 sec latency 145.289 ms 2024-05-08T20:27:13.495 INFO:tasks.workunit.client.1.smithi188.stdout: 10 154299 23.68 MB/sec execute 217 sec latency 106.564 ms 2024-05-08T20:27:13.769 INFO:tasks.workunit.client.0.smithi083.stdout: 10 281049 44.38 MB/sec execute 217 sec latency 164.586 ms 2024-05-08T20:27:14.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:14 smithi083 ceph-mon[38695]: pgmap v834: 129 pgs: 129 active+clean; 438 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 9.1 MiB/s wr, 417 op/s 2024-05-08T20:27:14.495 INFO:tasks.workunit.client.1.smithi188.stdout: 10 154679 23.70 MB/sec execute 218 sec latency 122.330 ms 2024-05-08T20:27:14.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:14 smithi188 ceph-mon[48761]: pgmap v834: 129 pgs: 129 active+clean; 438 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 9.1 MiB/s wr, 417 op/s 2024-05-08T20:27:14.769 INFO:tasks.workunit.client.0.smithi083.stdout: 10 281755 44.37 MB/sec execute 218 sec latency 295.986 ms 2024-05-08T20:27:15.498 INFO:tasks.workunit.client.1.smithi188.stdout: 10 155145 23.65 MB/sec execute 219 sec latency 74.383 ms 2024-05-08T20:27:15.769 INFO:tasks.workunit.client.0.smithi083.stdout: 10 283179 44.52 MB/sec execute 219 sec latency 117.466 ms 2024-05-08T20:27:16.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:16 smithi083 ceph-mon[38695]: pgmap v835: 129 pgs: 129 active+clean; 439 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 482 op/s 2024-05-08T20:27:16.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:16 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:27:16.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:16 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:27:16.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:16 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:27:16.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:16 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:27:16.498 INFO:tasks.workunit.client.1.smithi188.stdout: 10 155535 23.64 MB/sec execute 220 sec latency 92.009 ms 2024-05-08T20:27:16.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:16 smithi188 ceph-mon[48761]: pgmap v835: 129 pgs: 129 active+clean; 439 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 482 op/s 2024-05-08T20:27:16.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:16 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:27:16.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:16 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:27:16.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:16 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:27:16.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:16 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:27:16.769 INFO:tasks.workunit.client.0.smithi083.stdout: 10 284480 44.64 MB/sec execute 220 sec latency 120.458 ms 2024-05-08T20:27:17.498 INFO:tasks.workunit.client.1.smithi188.stdout: 10 156176 23.75 MB/sec execute 221 sec latency 54.633 ms 2024-05-08T20:27:17.769 INFO:tasks.workunit.client.0.smithi083.stdout: 10 284593 44.45 MB/sec execute 221 sec latency 130.550 ms 2024-05-08T20:27:18.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:18 smithi188 ceph-mon[48761]: pgmap v836: 129 pgs: 129 active+clean; 440 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 20 KiB/s rd, 10 MiB/s wr, 499 op/s 2024-05-08T20:27:18.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:18 smithi083 ceph-mon[38695]: pgmap v836: 129 pgs: 129 active+clean; 440 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 20 KiB/s rd, 10 MiB/s wr, 499 op/s 2024-05-08T20:27:18.498 INFO:tasks.workunit.client.1.smithi188.stdout: 10 156553 23.77 MB/sec execute 222 sec latency 99.759 ms 2024-05-08T20:27:18.769 INFO:tasks.workunit.client.0.smithi083.stdout: 10 285575 44.46 MB/sec execute 222 sec latency 122.456 ms 2024-05-08T20:27:19.498 INFO:tasks.workunit.client.1.smithi188.stdout: 10 156900 23.73 MB/sec execute 223 sec latency 106.013 ms 2024-05-08T20:27:19.769 INFO:tasks.workunit.client.0.smithi083.stdout: 10 286898 44.62 MB/sec execute 223 sec latency 141.574 ms 2024-05-08T20:27:20.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:20 smithi083 ceph-mon[38695]: pgmap v837: 129 pgs: 129 active+clean; 435 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 11 MiB/s wr, 535 op/s 2024-05-08T20:27:20.498 INFO:tasks.workunit.client.1.smithi188.stdout: 10 157278 23.68 MB/sec execute 224 sec latency 84.420 ms 2024-05-08T20:27:20.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:20 smithi188 ceph-mon[48761]: pgmap v837: 129 pgs: 129 active+clean; 435 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 11 MiB/s wr, 535 op/s 2024-05-08T20:27:20.769 INFO:tasks.workunit.client.0.smithi083.stdout: 10 288047 44.69 MB/sec execute 224 sec latency 132.095 ms 2024-05-08T20:27:21.499 INFO:tasks.workunit.client.1.smithi188.stdout: 10 157617 23.63 MB/sec execute 225 sec latency 96.968 ms 2024-05-08T20:27:21.769 INFO:tasks.workunit.client.0.smithi083.stdout: 10 288217 44.51 MB/sec execute 225 sec latency 108.605 ms 2024-05-08T20:27:22.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:22 smithi083 ceph-mon[38695]: pgmap v838: 129 pgs: 129 active+clean; 435 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 20 KiB/s rd, 13 MiB/s wr, 581 op/s 2024-05-08T20:27:22.498 INFO:tasks.workunit.client.1.smithi188.stdout: 10 157948 23.61 MB/sec execute 226 sec latency 126.357 ms 2024-05-08T20:27:22.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:22 smithi188 ceph-mon[48761]: pgmap v838: 129 pgs: 129 active+clean; 435 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 20 KiB/s rd, 13 MiB/s wr, 581 op/s 2024-05-08T20:27:22.769 INFO:tasks.workunit.client.0.smithi083.stdout: 10 289315 44.54 MB/sec execute 226 sec latency 145.182 ms 2024-05-08T20:27:23.500 INFO:tasks.workunit.client.1.smithi188.stdout: 10 158279 23.61 MB/sec execute 227 sec latency 93.450 ms 2024-05-08T20:27:23.769 INFO:tasks.workunit.client.0.smithi083.stdout: 10 290625 44.72 MB/sec execute 227 sec latency 137.460 ms 2024-05-08T20:27:24.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:24 smithi083 ceph-mon[38695]: pgmap v839: 129 pgs: 129 active+clean; 436 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 446 op/s 2024-05-08T20:27:24.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:27:24.500 INFO:tasks.workunit.client.1.smithi188.stdout: 10 158756 23.56 MB/sec execute 228 sec latency 77.910 ms 2024-05-08T20:27:24.598 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:24 smithi188 ceph-mon[48761]: pgmap v839: 129 pgs: 129 active+clean; 436 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 446 op/s 2024-05-08T20:27:24.598 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:24 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:27:24.769 INFO:tasks.workunit.client.0.smithi083.stdout: 10 291560 44.75 MB/sec execute 228 sec latency 338.180 ms 2024-05-08T20:27:25.500 INFO:tasks.workunit.client.1.smithi188.stdout: 10 159216 23.57 MB/sec execute 229 sec latency 71.902 ms 2024-05-08T20:27:25.769 INFO:tasks.workunit.client.0.smithi083.stdout: 10 291592 44.55 MB/sec execute 229 sec latency 533.581 ms 2024-05-08T20:27:26.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:26 smithi083 ceph-mon[38695]: pgmap v840: 129 pgs: 129 active+clean; 431 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 500 op/s 2024-05-08T20:27:26.500 INFO:tasks.workunit.client.1.smithi188.stdout: 10 159821 23.68 MB/sec execute 230 sec latency 77.169 ms 2024-05-08T20:27:26.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:26 smithi188 ceph-mon[48761]: pgmap v840: 129 pgs: 129 active+clean; 431 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 500 op/s 2024-05-08T20:27:26.769 INFO:tasks.workunit.client.0.smithi083.stdout: 10 291611 44.36 MB/sec execute 230 sec latency 725.870 ms 2024-05-08T20:27:27.500 INFO:tasks.workunit.client.1.smithi188.stdout: 10 160161 23.68 MB/sec execute 231 sec latency 115.645 ms 2024-05-08T20:27:27.769 INFO:tasks.workunit.client.0.smithi083.stdout: 10 291615 44.17 MB/sec execute 231 sec latency 658.253 ms 2024-05-08T20:27:28.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:28 smithi083 ceph-mon[38695]: pgmap v841: 129 pgs: 129 active+clean; 428 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 10 MiB/s wr, 510 op/s 2024-05-08T20:27:28.500 INFO:tasks.workunit.client.1.smithi188.stdout: 10 160437 23.63 MB/sec execute 232 sec latency 134.888 ms 2024-05-08T20:27:28.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:28 smithi188 ceph-mon[48761]: pgmap v841: 129 pgs: 129 active+clean; 428 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 10 MiB/s wr, 510 op/s 2024-05-08T20:27:28.769 INFO:tasks.workunit.client.0.smithi083.stdout: 10 291648 43.98 MB/sec execute 232 sec latency 444.429 ms 2024-05-08T20:27:29.500 INFO:tasks.workunit.client.1.smithi188.stdout: 10 160812 23.58 MB/sec execute 233 sec latency 67.679 ms 2024-05-08T20:27:29.769 INFO:tasks.workunit.client.0.smithi083.stdout: 10 291780 43.81 MB/sec execute 233 sec latency 443.062 ms 2024-05-08T20:27:30.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:30 smithi083 ceph-mon[38695]: pgmap v842: 129 pgs: 129 active+clean; 432 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 9.9 MiB/s wr, 469 op/s 2024-05-08T20:27:30.500 INFO:tasks.workunit.client.1.smithi188.stdout: 10 161178 23.54 MB/sec execute 234 sec latency 116.982 ms 2024-05-08T20:27:30.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:30 smithi188 ceph-mon[48761]: pgmap v842: 129 pgs: 129 active+clean; 432 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 9.9 MiB/s wr, 469 op/s 2024-05-08T20:27:30.769 INFO:tasks.workunit.client.0.smithi083.stdout: 10 293046 43.86 MB/sec execute 234 sec latency 123.523 ms 2024-05-08T20:27:31.500 INFO:tasks.workunit.client.1.smithi188.stdout: 10 161592 23.58 MB/sec execute 235 sec latency 105.612 ms 2024-05-08T20:27:31.769 INFO:tasks.workunit.client.0.smithi083.stdout: 10 294190 44.02 MB/sec execute 235 sec latency 138.043 ms 2024-05-08T20:27:32.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:32 smithi083 ceph-mon[38695]: pgmap v843: 129 pgs: 129 active+clean; 438 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 503 op/s 2024-05-08T20:27:32.500 INFO:tasks.workunit.client.1.smithi188.stdout: 10 161962 23.54 MB/sec execute 236 sec latency 109.040 ms 2024-05-08T20:27:32.598 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:32 smithi188 ceph-mon[48761]: pgmap v843: 129 pgs: 129 active+clean; 438 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 503 op/s 2024-05-08T20:27:32.770 INFO:tasks.workunit.client.0.smithi083.stdout: 10 295164 44.05 MB/sec execute 236 sec latency 116.772 ms 2024-05-08T20:27:33.500 INFO:tasks.workunit.client.1.smithi188.stdout: 10 162405 23.50 MB/sec execute 237 sec latency 69.226 ms 2024-05-08T20:27:33.770 INFO:tasks.workunit.client.0.smithi083.stdout: 10 295382 43.88 MB/sec execute 237 sec latency 113.019 ms 2024-05-08T20:27:34.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:34 smithi083 ceph-mon[38695]: pgmap v844: 129 pgs: 129 active+clean; 435 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 8.4 MiB/s wr, 430 op/s 2024-05-08T20:27:34.500 INFO:tasks.workunit.client.1.smithi188.stdout: 10 162853 23.55 MB/sec execute 238 sec latency 96.898 ms 2024-05-08T20:27:34.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:34 smithi188 ceph-mon[48761]: pgmap v844: 129 pgs: 129 active+clean; 435 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 8.4 MiB/s wr, 430 op/s 2024-05-08T20:27:34.770 INFO:tasks.workunit.client.0.smithi083.stdout: 10 296846 43.96 MB/sec execute 238 sec latency 137.841 ms 2024-05-08T20:27:35.500 INFO:tasks.workunit.client.1.smithi188.stdout: 10 163355 23.57 MB/sec execute 239 sec latency 97.716 ms 2024-05-08T20:27:35.770 INFO:tasks.workunit.client.0.smithi083.stdout: 10 298207 44.18 MB/sec execute 239 sec latency 123.106 ms 2024-05-08T20:27:36.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:36 smithi083 ceph-mon[38695]: pgmap v845: 129 pgs: 129 active+clean; 436 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 9.8 MiB/s wr, 496 op/s 2024-05-08T20:27:36.500 INFO:tasks.workunit.client.1.smithi188.stdout: 10 163807 23.62 MB/sec execute 240 sec latency 63.474 ms 2024-05-08T20:27:36.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:36 smithi188 ceph-mon[48761]: pgmap v845: 129 pgs: 129 active+clean; 436 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 9.8 MiB/s wr, 496 op/s 2024-05-08T20:27:36.770 INFO:tasks.workunit.client.0.smithi083.stdout: 10 298727 44.11 MB/sec execute 240 sec latency 138.033 ms 2024-05-08T20:27:37.500 INFO:tasks.workunit.client.1.smithi188.stdout: 10 164169 23.57 MB/sec execute 241 sec latency 124.908 ms 2024-05-08T20:27:37.770 INFO:tasks.workunit.client.0.smithi083.stdout: 10 298938 43.95 MB/sec execute 241 sec latency 133.758 ms 2024-05-08T20:27:38.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:38 smithi083 ceph-mon[38695]: pgmap v846: 129 pgs: 129 active+clean; 434 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 529 op/s 2024-05-08T20:27:38.501 INFO:tasks.workunit.client.1.smithi188.stdout: 10 164538 23.53 MB/sec execute 242 sec latency 136.623 ms 2024-05-08T20:27:38.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:38 smithi188 ceph-mon[48761]: pgmap v846: 129 pgs: 129 active+clean; 434 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 529 op/s 2024-05-08T20:27:38.770 INFO:tasks.workunit.client.0.smithi083.stdout: 10 300388 44.03 MB/sec execute 242 sec latency 160.607 ms 2024-05-08T20:27:39.501 INFO:tasks.workunit.client.1.smithi188.stdout: 10 164934 23.52 MB/sec execute 243 sec latency 84.257 ms 2024-05-08T20:27:39.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:27:39.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:27:39.770 INFO:tasks.workunit.client.0.smithi083.stdout: 10 301844 44.26 MB/sec execute 243 sec latency 138.063 ms 2024-05-08T20:27:40.501 INFO:tasks.workunit.client.1.smithi188.stdout: 10 165259 23.51 MB/sec execute 244 sec latency 153.500 ms 2024-05-08T20:27:40.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:40 smithi188 ceph-mon[48761]: pgmap v847: 129 pgs: 129 active+clean; 434 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 478 op/s 2024-05-08T20:27:40.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:40 smithi083 ceph-mon[38695]: pgmap v847: 129 pgs: 129 active+clean; 434 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 478 op/s 2024-05-08T20:27:40.770 INFO:tasks.workunit.client.0.smithi083.stdout: 10 302276 44.17 MB/sec execute 244 sec latency 317.650 ms 2024-05-08T20:27:41.501 INFO:tasks.workunit.client.1.smithi188.stdout: 10 165693 23.47 MB/sec execute 245 sec latency 63.685 ms 2024-05-08T20:27:41.770 INFO:tasks.workunit.client.0.smithi083.stdout: 10 302606 44.05 MB/sec execute 245 sec latency 125.563 ms 2024-05-08T20:27:42.501 INFO:tasks.workunit.client.1.smithi188.stdout: 10 166123 23.45 MB/sec execute 246 sec latency 84.959 ms 2024-05-08T20:27:42.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:42 smithi188 ceph-mon[48761]: pgmap v848: 129 pgs: 129 active+clean; 434 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 541 op/s 2024-05-08T20:27:42.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:42 smithi083 ceph-mon[38695]: pgmap v848: 129 pgs: 129 active+clean; 434 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 541 op/s 2024-05-08T20:27:42.770 INFO:tasks.workunit.client.0.smithi083.stdout: 10 304033 44.13 MB/sec execute 246 sec latency 143.468 ms 2024-05-08T20:27:43.501 INFO:tasks.workunit.client.1.smithi188.stdout: 10 166659 23.53 MB/sec execute 247 sec latency 83.700 ms 2024-05-08T20:27:43.770 INFO:tasks.workunit.client.0.smithi083.stdout: 10 305743 44.40 MB/sec execute 247 sec latency 116.853 ms 2024-05-08T20:27:44.501 INFO:tasks.workunit.client.1.smithi188.stdout: 10 167068 23.52 MB/sec execute 248 sec latency 76.514 ms 2024-05-08T20:27:44.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:44 smithi188 ceph-mon[48761]: pgmap v849: 129 pgs: 129 active+clean; 433 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 8.7 MiB/s wr, 403 op/s 2024-05-08T20:27:44.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:44 smithi083 ceph-mon[38695]: pgmap v849: 129 pgs: 129 active+clean; 433 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 8.7 MiB/s wr, 403 op/s 2024-05-08T20:27:44.770 INFO:tasks.workunit.client.0.smithi083.stdout: 10 305869 44.23 MB/sec execute 248 sec latency 122.391 ms 2024-05-08T20:27:45.501 INFO:tasks.workunit.client.1.smithi188.stdout: 10 167491 23.55 MB/sec execute 249 sec latency 103.618 ms 2024-05-08T20:27:45.770 INFO:tasks.workunit.client.0.smithi083.stdout: 10 306155 44.11 MB/sec execute 249 sec latency 264.947 ms 2024-05-08T20:27:46.501 INFO:tasks.workunit.client.1.smithi188.stdout: 10 167835 23.52 MB/sec execute 250 sec latency 172.303 ms 2024-05-08T20:27:46.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:46 smithi188 ceph-mon[48761]: pgmap v850: 129 pgs: 129 active+clean; 429 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 486 op/s 2024-05-08T20:27:46.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:46 smithi083 ceph-mon[38695]: pgmap v850: 129 pgs: 129 active+clean; 429 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 486 op/s 2024-05-08T20:27:46.770 INFO:tasks.workunit.client.0.smithi083.stdout: 10 307746 44.26 MB/sec execute 250 sec latency 121.734 ms 2024-05-08T20:27:47.501 INFO:tasks.workunit.client.1.smithi188.stdout: 10 168335 23.50 MB/sec execute 251 sec latency 32.733 ms 2024-05-08T20:27:47.776 INFO:tasks.workunit.client.0.smithi083.stdout: 10 309292 44.45 MB/sec execute 251 sec latency 127.583 ms 2024-05-08T20:27:48.379 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:48 smithi083 ceph-mon[38695]: pgmap v851: 129 pgs: 129 active+clean; 434 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 500 op/s 2024-05-08T20:27:48.501 INFO:tasks.workunit.client.1.smithi188.stdout: 10 168740 23.52 MB/sec execute 252 sec latency 96.049 ms 2024-05-08T20:27:48.560 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:48 smithi188 ceph-mon[48761]: pgmap v851: 129 pgs: 129 active+clean; 434 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 500 op/s 2024-05-08T20:27:48.773 INFO:tasks.workunit.client.0.smithi083.stdout: 10 309417 44.29 MB/sec execute 252 sec latency 124.865 ms 2024-05-08T20:27:49.501 INFO:tasks.workunit.client.1.smithi188.stdout: 10 169044 23.47 MB/sec execute 253 sec latency 164.855 ms 2024-05-08T20:27:49.773 INFO:tasks.workunit.client.0.smithi083.stdout: 10 310287 44.30 MB/sec execute 253 sec latency 144.340 ms 2024-05-08T20:27:50.501 INFO:tasks.workunit.client.1.smithi188.stdout: 10 169481 23.42 MB/sec execute 254 sec latency 97.966 ms 2024-05-08T20:27:50.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:50 smithi188 ceph-mon[48761]: pgmap v852: 129 pgs: 129 active+clean; 440 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 480 op/s 2024-05-08T20:27:50.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:50 smithi083 ceph-mon[38695]: pgmap v852: 129 pgs: 129 active+clean; 440 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 480 op/s 2024-05-08T20:27:50.773 INFO:tasks.workunit.client.0.smithi083.stdout: 10 311295 44.32 MB/sec execute 254 sec latency 285.279 ms 2024-05-08T20:27:51.501 INFO:tasks.workunit.client.1.smithi188.stdout: 10 169905 23.44 MB/sec execute 255 sec latency 73.582 ms 2024-05-08T20:27:51.773 INFO:tasks.workunit.client.0.smithi083.stdout: 10 312836 44.51 MB/sec execute 255 sec latency 248.528 ms 2024-05-08T20:27:52.514 INFO:tasks.workunit.client.1.smithi188.stdout: 10 170405 23.49 MB/sec execute 256 sec latency 123.125 ms 2024-05-08T20:27:52.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:52 smithi188 ceph-mon[48761]: pgmap v853: 129 pgs: 129 active+clean; 430 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 564 op/s 2024-05-08T20:27:52.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:52 smithi083 ceph-mon[38695]: pgmap v853: 129 pgs: 129 active+clean; 430 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 564 op/s 2024-05-08T20:27:52.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 312960 44.34 MB/sec execute 256 sec latency 143.330 ms 2024-05-08T20:27:53.514 INFO:tasks.workunit.client.1.smithi188.stdout: 10 170785 23.52 MB/sec execute 257 sec latency 67.983 ms 2024-05-08T20:27:53.773 INFO:tasks.workunit.client.0.smithi083.stdout: 10 313829 44.36 MB/sec execute 257 sec latency 140.161 ms 2024-05-08T20:27:54.514 INFO:tasks.workunit.client.1.smithi188.stdout: 10 171097 23.48 MB/sec execute 258 sec latency 103.547 ms 2024-05-08T20:27:54.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:54 smithi188 ceph-mon[48761]: pgmap v854: 129 pgs: 129 active+clean; 436 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 14 KiB/s rd, 9.0 MiB/s wr, 470 op/s 2024-05-08T20:27:54.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:27:54.698 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:54 smithi083 ceph-mon[38695]: pgmap v854: 129 pgs: 129 active+clean; 436 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 14 KiB/s rd, 9.0 MiB/s wr, 470 op/s 2024-05-08T20:27:54.698 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:27:54.773 INFO:tasks.workunit.client.0.smithi083.stdout: 10 315114 44.47 MB/sec execute 258 sec latency 142.727 ms 2024-05-08T20:27:55.514 INFO:tasks.workunit.client.1.smithi188.stdout: 10 171465 23.44 MB/sec execute 259 sec latency 86.928 ms 2024-05-08T20:27:55.773 INFO:tasks.workunit.client.0.smithi083.stdout: 10 316387 44.56 MB/sec execute 259 sec latency 203.170 ms 2024-05-08T20:27:56.514 INFO:tasks.workunit.client.1.smithi188.stdout: 10 171885 23.42 MB/sec execute 260 sec latency 86.436 ms 2024-05-08T20:27:56.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:56 smithi188 ceph-mon[48761]: pgmap v855: 129 pgs: 129 active+clean; 437 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 504 op/s 2024-05-08T20:27:56.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:56 smithi083 ceph-mon[38695]: pgmap v855: 129 pgs: 129 active+clean; 437 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 504 op/s 2024-05-08T20:27:56.773 INFO:tasks.workunit.client.0.smithi083.stdout: 10 316509 44.40 MB/sec execute 260 sec latency 311.619 ms 2024-05-08T20:27:57.514 INFO:tasks.workunit.client.1.smithi188.stdout: 10 172258 23.41 MB/sec execute 261 sec latency 79.708 ms 2024-05-08T20:27:57.773 INFO:tasks.workunit.client.0.smithi083.stdout: 10 317282 44.41 MB/sec execute 261 sec latency 136.601 ms 2024-05-08T20:27:58.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:27:58 smithi083 ceph-mon[38695]: pgmap v856: 129 pgs: 129 active+clean; 437 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 479 op/s 2024-05-08T20:27:58.515 INFO:tasks.workunit.client.1.smithi188.stdout: 10 172621 23.40 MB/sec execute 262 sec latency 133.144 ms 2024-05-08T20:27:58.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:27:58 smithi188 ceph-mon[48761]: pgmap v856: 129 pgs: 129 active+clean; 437 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 479 op/s 2024-05-08T20:27:58.773 INFO:tasks.workunit.client.0.smithi083.stdout: 10 318641 44.51 MB/sec execute 262 sec latency 119.741 ms 2024-05-08T20:27:59.515 INFO:tasks.workunit.client.1.smithi188.stdout: 10 173056 23.36 MB/sec execute 263 sec latency 105.148 ms 2024-05-08T20:27:59.773 INFO:tasks.workunit.client.0.smithi083.stdout: 10 319942 44.61 MB/sec execute 263 sec latency 116.578 ms 2024-05-08T20:28:00.515 INFO:tasks.workunit.client.1.smithi188.stdout: 10 173468 23.37 MB/sec execute 264 sec latency 121.662 ms 2024-05-08T20:28:00.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:00 smithi188 ceph-mon[48761]: pgmap v857: 129 pgs: 129 active+clean; 438 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 441 op/s 2024-05-08T20:28:00.695 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:00 smithi083 ceph-mon[38695]: pgmap v857: 129 pgs: 129 active+clean; 438 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 441 op/s 2024-05-08T20:28:00.773 INFO:tasks.workunit.client.0.smithi083.stdout: 10 320055 44.45 MB/sec execute 264 sec latency 130.314 ms 2024-05-08T20:28:01.515 INFO:tasks.workunit.client.1.smithi188.stdout: 10 174058 23.44 MB/sec execute 265 sec latency 84.930 ms 2024-05-08T20:28:01.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 320096 44.29 MB/sec execute 265 sec latency 472.695 ms 2024-05-08T20:28:02.515 INFO:tasks.workunit.client.1.smithi188.stdout: 10 174470 23.46 MB/sec execute 266 sec latency 77.521 ms 2024-05-08T20:28:02.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:02 smithi188 ceph-mon[48761]: pgmap v858: 129 pgs: 129 active+clean; 437 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 20 KiB/s rd, 11 MiB/s wr, 548 op/s 2024-05-08T20:28:02.698 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:02 smithi083 ceph-mon[38695]: pgmap v858: 129 pgs: 129 active+clean; 437 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 20 KiB/s rd, 11 MiB/s wr, 548 op/s 2024-05-08T20:28:02.773 INFO:tasks.workunit.client.0.smithi083.stdout: 10 320151 44.12 MB/sec execute 266 sec latency 560.003 ms 2024-05-08T20:28:03.515 INFO:tasks.workunit.client.1.smithi188.stdout: 10 174805 23.43 MB/sec execute 267 sec latency 98.082 ms 2024-05-08T20:28:03.773 INFO:tasks.workunit.client.0.smithi083.stdout: 10 320205 43.96 MB/sec execute 267 sec latency 599.909 ms 2024-05-08T20:28:04.515 INFO:tasks.workunit.client.1.smithi188.stdout: 10 175203 23.41 MB/sec execute 268 sec latency 89.573 ms 2024-05-08T20:28:04.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:04 smithi188 ceph-mon[48761]: pgmap v859: 129 pgs: 129 active+clean; 436 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 442 op/s 2024-05-08T20:28:04.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:04 smithi083 ceph-mon[38695]: pgmap v859: 129 pgs: 129 active+clean; 436 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 442 op/s 2024-05-08T20:28:04.773 INFO:tasks.workunit.client.0.smithi083.stdout: 10 320516 43.88 MB/sec execute 268 sec latency 464.804 ms 2024-05-08T20:28:05.515 INFO:tasks.workunit.client.1.smithi188.stdout: 10 175582 23.38 MB/sec execute 269 sec latency 113.241 ms 2024-05-08T20:28:05.773 INFO:tasks.workunit.client.0.smithi083.stdout: 10 321536 43.85 MB/sec execute 269 sec latency 545.738 ms 2024-05-08T20:28:06.516 INFO:tasks.workunit.client.1.smithi188.stdout: 10 175997 23.39 MB/sec execute 270 sec latency 110.028 ms 2024-05-08T20:28:06.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:06 smithi188 ceph-mon[48761]: pgmap v860: 129 pgs: 129 active+clean; 435 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 9.7 MiB/s wr, 408 op/s 2024-05-08T20:28:06.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:06 smithi083 ceph-mon[38695]: pgmap v860: 129 pgs: 129 active+clean; 435 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 9.7 MiB/s wr, 408 op/s 2024-05-08T20:28:06.773 INFO:tasks.workunit.client.0.smithi083.stdout: 10 322562 43.98 MB/sec execute 270 sec latency 168.201 ms 2024-05-08T20:28:07.516 INFO:tasks.workunit.client.1.smithi188.stdout: 10 176432 23.35 MB/sec execute 271 sec latency 157.448 ms 2024-05-08T20:28:07.773 INFO:tasks.workunit.client.0.smithi083.stdout: 10 323535 44.00 MB/sec execute 271 sec latency 133.823 ms 2024-05-08T20:28:08.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:08 smithi083 ceph-mon[38695]: pgmap v861: 129 pgs: 129 active+clean; 448 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 525 op/s 2024-05-08T20:28:08.516 INFO:tasks.workunit.client.1.smithi188.stdout: 10 176900 23.36 MB/sec execute 272 sec latency 93.692 ms 2024-05-08T20:28:08.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:08 smithi188 ceph-mon[48761]: pgmap v861: 129 pgs: 129 active+clean; 448 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 525 op/s 2024-05-08T20:28:08.773 INFO:tasks.workunit.client.0.smithi083.stdout: 10 323752 43.86 MB/sec execute 272 sec latency 122.765 ms 2024-05-08T20:28:09.262 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:28:09.516 INFO:tasks.workunit.client.1.smithi188.stdout: 10 177362 23.41 MB/sec execute 273 sec latency 98.539 ms 2024-05-08T20:28:09.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:28:09.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 325441 44.02 MB/sec execute 273 sec latency 119.433 ms 2024-05-08T20:28:10.516 INFO:tasks.workunit.client.1.smithi188.stdout: 10 177822 23.42 MB/sec execute 274 sec latency 71.574 ms 2024-05-08T20:28:10.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:10 smithi188 ceph-mon[48761]: pgmap v862: 129 pgs: 129 active+clean; 439 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 9.8 MiB/s wr, 520 op/s 2024-05-08T20:28:10.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:10 smithi083 ceph-mon[38695]: pgmap v862: 129 pgs: 129 active+clean; 439 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 9.8 MiB/s wr, 520 op/s 2024-05-08T20:28:10.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 326967 44.21 MB/sec execute 274 sec latency 201.930 ms 2024-05-08T20:28:11.516 INFO:tasks.workunit.client.1.smithi188.stdout: 10 178207 23.43 MB/sec execute 275 sec latency 138.808 ms 2024-05-08T20:28:11.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:11 smithi188 ceph-mon[48761]: pgmap v863: 129 pgs: 129 active+clean; 437 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 20 KiB/s rd, 12 MiB/s wr, 576 op/s 2024-05-08T20:28:11.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:11 smithi083 ceph-mon[38695]: pgmap v863: 129 pgs: 129 active+clean; 437 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 20 KiB/s rd, 12 MiB/s wr, 576 op/s 2024-05-08T20:28:11.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 327142 44.06 MB/sec execute 275 sec latency 141.548 ms 2024-05-08T20:28:12.516 INFO:tasks.workunit.client.1.smithi188.stdout: 10 178509 23.38 MB/sec execute 276 sec latency 124.382 ms 2024-05-08T20:28:12.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 328102 44.08 MB/sec execute 276 sec latency 117.890 ms 2024-05-08T20:28:13.516 INFO:tasks.workunit.client.1.smithi188.stdout: 10 178934 23.36 MB/sec execute 277 sec latency 44.355 ms 2024-05-08T20:28:13.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:13 smithi083 ceph-mon[38695]: pgmap v864: 129 pgs: 129 active+clean; 436 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 400 op/s 2024-05-08T20:28:13.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 329451 44.20 MB/sec execute 277 sec latency 127.464 ms 2024-05-08T20:28:13.846 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:13 smithi188 ceph-mon[48761]: pgmap v864: 129 pgs: 129 active+clean; 436 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 400 op/s 2024-05-08T20:28:14.516 INFO:tasks.workunit.client.1.smithi188.stdout: 10 179267 23.33 MB/sec execute 278 sec latency 120.046 ms 2024-05-08T20:28:14.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 330615 44.27 MB/sec execute 278 sec latency 129.704 ms 2024-05-08T20:28:15.516 INFO:tasks.workunit.client.1.smithi188.stdout: 10 179663 23.34 MB/sec execute 279 sec latency 163.148 ms 2024-05-08T20:28:15.698 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:15 smithi083 ceph-mon[38695]: pgmap v865: 129 pgs: 129 active+clean; 437 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 446 op/s 2024-05-08T20:28:15.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 330711 44.12 MB/sec execute 279 sec latency 233.209 ms 2024-05-08T20:28:15.846 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:15 smithi188 ceph-mon[48761]: pgmap v865: 129 pgs: 129 active+clean; 437 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 446 op/s 2024-05-08T20:28:16.516 INFO:tasks.workunit.client.1.smithi188.stdout: 10 180097 23.31 MB/sec execute 280 sec latency 104.535 ms 2024-05-08T20:28:16.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:16 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:28:16.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 331868 44.14 MB/sec execute 280 sec latency 246.291 ms 2024-05-08T20:28:16.846 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:16 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:28:17.519 INFO:tasks.workunit.client.1.smithi188.stdout: 10 180507 23.30 MB/sec execute 281 sec latency 66.036 ms 2024-05-08T20:28:17.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:17 smithi083 ceph-mon[38695]: pgmap v866: 129 pgs: 129 active+clean; 433 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 571 op/s 2024-05-08T20:28:17.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 333141 44.28 MB/sec execute 281 sec latency 126.191 ms 2024-05-08T20:28:17.846 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:17 smithi188 ceph-mon[48761]: pgmap v866: 129 pgs: 129 active+clean; 433 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 571 op/s 2024-05-08T20:28:18.519 INFO:tasks.workunit.client.1.smithi188.stdout: 10 181047 23.36 MB/sec execute 282 sec latency 61.155 ms 2024-05-08T20:28:18.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 334162 44.32 MB/sec execute 282 sec latency 135.840 ms 2024-05-08T20:28:19.519 INFO:tasks.workunit.client.1.smithi188.stdout: 10 181403 23.35 MB/sec execute 283 sec latency 111.429 ms 2024-05-08T20:28:19.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 334333 44.18 MB/sec execute 283 sec latency 112.052 ms 2024-05-08T20:28:20.196 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:19 smithi083 ceph-mon[38695]: pgmap v867: 129 pgs: 129 active+clean; 430 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 14 KiB/s rd, 9.6 MiB/s wr, 450 op/s 2024-05-08T20:28:20.196 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:19 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:28:20.196 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:19 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:28:20.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:19 smithi188 ceph-mon[48761]: pgmap v867: 129 pgs: 129 active+clean; 430 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 14 KiB/s rd, 9.6 MiB/s wr, 450 op/s 2024-05-08T20:28:20.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:19 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:28:20.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:19 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:28:20.519 INFO:tasks.workunit.client.1.smithi188.stdout: 10 181882 23.37 MB/sec execute 284 sec latency 85.109 ms 2024-05-08T20:28:20.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 335411 44.20 MB/sec execute 284 sec latency 135.475 ms 2024-05-08T20:28:21.519 INFO:tasks.workunit.client.1.smithi188.stdout: 10 182205 23.33 MB/sec execute 285 sec latency 133.054 ms 2024-05-08T20:28:21.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 336653 44.32 MB/sec execute 285 sec latency 234.834 ms 2024-05-08T20:28:22.197 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:21 smithi083 ceph-mon[38695]: pgmap v868: 129 pgs: 129 active+clean; 435 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 20 KiB/s rd, 12 MiB/s wr, 556 op/s 2024-05-08T20:28:22.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:21 smithi188 ceph-mon[48761]: pgmap v868: 129 pgs: 129 active+clean; 435 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 20 KiB/s rd, 12 MiB/s wr, 556 op/s 2024-05-08T20:28:22.519 INFO:tasks.workunit.client.1.smithi188.stdout: 10 182474 23.29 MB/sec execute 286 sec latency 132.414 ms 2024-05-08T20:28:22.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 337706 44.37 MB/sec execute 286 sec latency 131.114 ms 2024-05-08T20:28:23.519 INFO:tasks.workunit.client.1.smithi188.stdout: 10 182860 23.27 MB/sec execute 287 sec latency 34.273 ms 2024-05-08T20:28:23.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 337832 44.22 MB/sec execute 287 sec latency 128.213 ms 2024-05-08T20:28:24.196 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:23 smithi083 ceph-mon[38695]: pgmap v869: 129 pgs: 129 active+clean; 437 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 442 op/s 2024-05-08T20:28:24.196 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:23 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:28:24.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:23 smithi188 ceph-mon[48761]: pgmap v869: 129 pgs: 129 active+clean; 437 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 442 op/s 2024-05-08T20:28:24.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:23 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:28:24.521 INFO:tasks.workunit.client.1.smithi188.stdout: 10 183187 23.27 MB/sec execute 288 sec latency 148.900 ms 2024-05-08T20:28:24.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 338961 44.25 MB/sec execute 288 sec latency 132.366 ms 2024-05-08T20:28:25.521 INFO:tasks.workunit.client.1.smithi188.stdout: 10 183571 23.24 MB/sec execute 289 sec latency 93.706 ms 2024-05-08T20:28:25.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 340202 44.37 MB/sec execute 289 sec latency 137.483 ms 2024-05-08T20:28:26.196 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:25 smithi083 ceph-mon[38695]: pgmap v870: 129 pgs: 129 active+clean; 435 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 531 op/s 2024-05-08T20:28:26.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:25 smithi188 ceph-mon[48761]: pgmap v870: 129 pgs: 129 active+clean; 435 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 531 op/s 2024-05-08T20:28:26.521 INFO:tasks.workunit.client.1.smithi188.stdout: 10 184035 23.24 MB/sec execute 290 sec latency 108.812 ms 2024-05-08T20:28:26.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 341201 44.41 MB/sec execute 290 sec latency 352.421 ms 2024-05-08T20:28:27.522 INFO:tasks.workunit.client.1.smithi188.stdout: 10 184562 23.28 MB/sec execute 291 sec latency 63.483 ms 2024-05-08T20:28:27.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 341320 44.27 MB/sec execute 291 sec latency 413.022 ms 2024-05-08T20:28:28.350 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:27 smithi188 ceph-mon[48761]: pgmap v871: 129 pgs: 129 active+clean; 428 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 20 KiB/s rd, 10 MiB/s wr, 540 op/s 2024-05-08T20:28:28.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:27 smithi083 ceph-mon[38695]: pgmap v871: 129 pgs: 129 active+clean; 428 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 20 KiB/s rd, 10 MiB/s wr, 540 op/s 2024-05-08T20:28:28.521 INFO:tasks.workunit.client.1.smithi188.stdout: 10 184916 23.27 MB/sec execute 292 sec latency 150.203 ms 2024-05-08T20:28:28.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 341788 44.20 MB/sec execute 292 sec latency 120.958 ms 2024-05-08T20:28:29.522 INFO:tasks.workunit.client.1.smithi188.stdout: 10 185341 23.29 MB/sec execute 293 sec latency 53.089 ms 2024-05-08T20:28:29.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 343166 44.28 MB/sec execute 293 sec latency 137.287 ms 2024-05-08T20:28:30.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:29 smithi188 ceph-mon[48761]: pgmap v872: 129 pgs: 129 active+clean; 429 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 9.6 MiB/s wr, 480 op/s 2024-05-08T20:28:30.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:29 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:28:30.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:29 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:28:30.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:29 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:28:30.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:29 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:28:30.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:29 smithi083 ceph-mon[38695]: pgmap v872: 129 pgs: 129 active+clean; 429 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 9.6 MiB/s wr, 480 op/s 2024-05-08T20:28:30.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:29 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:28:30.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:29 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:28:30.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:29 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:28:30.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:29 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:28:30.522 INFO:tasks.workunit.client.1.smithi188.stdout: 10 185631 23.25 MB/sec execute 294 sec latency 132.720 ms 2024-05-08T20:28:30.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 344745 44.46 MB/sec execute 294 sec latency 142.685 ms 2024-05-08T20:28:31.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:30 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:28:31.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:30 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:28:31.522 INFO:tasks.workunit.client.1.smithi188.stdout: 10 186107 23.24 MB/sec execute 295 sec latency 27.805 ms 2024-05-08T20:28:31.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 344872 44.31 MB/sec execute 295 sec latency 118.981 ms 2024-05-08T20:28:32.348 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:31 smithi188 ceph-mon[48761]: pgmap v873: 129 pgs: 129 active+clean; 436 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 504 op/s 2024-05-08T20:28:32.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:31 smithi083 ceph-mon[38695]: pgmap v873: 129 pgs: 129 active+clean; 436 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 504 op/s 2024-05-08T20:28:32.522 INFO:tasks.workunit.client.1.smithi188.stdout: 10 186438 23.21 MB/sec execute 296 sec latency 122.688 ms 2024-05-08T20:28:32.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 345645 44.32 MB/sec execute 296 sec latency 159.445 ms 2024-05-08T20:28:33.522 INFO:tasks.workunit.client.1.smithi188.stdout: 10 186782 23.22 MB/sec execute 297 sec latency 114.446 ms 2024-05-08T20:28:33.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 347025 44.42 MB/sec execute 297 sec latency 122.491 ms 2024-05-08T20:28:34.306 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:33 smithi188 ceph-mon[48761]: pgmap v874: 129 pgs: 129 active+clean; 439 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 357 op/s 2024-05-08T20:28:34.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:33 smithi083 ceph-mon[38695]: pgmap v874: 129 pgs: 129 active+clean; 439 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 357 op/s 2024-05-08T20:28:34.523 INFO:tasks.workunit.client.1.smithi188.stdout: 10 187112 23.18 MB/sec execute 298 sec latency 94.078 ms 2024-05-08T20:28:34.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 348306 44.50 MB/sec execute 298 sec latency 130.998 ms 2024-05-08T20:28:35.522 INFO:tasks.workunit.client.1.smithi188.stdout: 10 187535 23.16 MB/sec execute 299 sec latency 92.107 ms 2024-05-08T20:28:35.774 INFO:tasks.workunit.client.0.smithi083.stdout: 10 348435 44.36 MB/sec execute 299 sec latency 107.795 ms 2024-05-08T20:28:36.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:35 smithi188 ceph-mon[48761]: pgmap v875: 129 pgs: 129 active+clean; 432 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 445 op/s 2024-05-08T20:28:36.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:35 smithi083 ceph-mon[38695]: pgmap v875: 129 pgs: 129 active+clean; 432 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 445 op/s 2024-05-08T20:28:36.522 INFO:tasks.workunit.client.1.smithi188.stdout: 10 188055 23.22 MB/sec execute 300 sec latency 81.666 ms 2024-05-08T20:28:36.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 349594 44.39 MB/sec execute 300 sec latency 125.257 ms 2024-05-08T20:28:37.522 INFO:tasks.workunit.client.1.smithi188.stdout: 10 188505 23.22 MB/sec execute 301 sec latency 73.801 ms 2024-05-08T20:28:37.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 350257 44.37 MB/sec execute 301 sec latency 362.815 ms 2024-05-08T20:28:38.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:38 smithi188 ceph-mon[48761]: pgmap v876: 129 pgs: 129 active+clean; 426 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 9.4 MiB/s wr, 453 op/s 2024-05-08T20:28:38.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:37 smithi083 ceph-mon[38695]: pgmap v876: 129 pgs: 129 active+clean; 426 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 9.4 MiB/s wr, 453 op/s 2024-05-08T20:28:38.522 INFO:tasks.workunit.client.1.smithi188.stdout: 10 188892 23.23 MB/sec execute 302 sec latency 120.834 ms 2024-05-08T20:28:38.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 350547 44.31 MB/sec execute 302 sec latency 483.989 ms 2024-05-08T20:28:39.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:28:39.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:28:39.522 INFO:tasks.workunit.client.1.smithi188.stdout: 10 189212 23.21 MB/sec execute 303 sec latency 78.770 ms 2024-05-08T20:28:39.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 350639 44.18 MB/sec execute 303 sec latency 621.636 ms 2024-05-08T20:28:40.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:40 smithi188 ceph-mon[48761]: pgmap v877: 129 pgs: 129 active+clean; 423 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 453 op/s 2024-05-08T20:28:40.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:40 smithi083 ceph-mon[38695]: pgmap v877: 129 pgs: 129 active+clean; 423 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 453 op/s 2024-05-08T20:28:40.522 INFO:tasks.workunit.client.1.smithi188.stdout: 10 189629 23.19 MB/sec execute 304 sec latency 80.644 ms 2024-05-08T20:28:40.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 350835 44.07 MB/sec execute 304 sec latency 831.790 ms 2024-05-08T20:28:41.522 INFO:tasks.workunit.client.1.smithi188.stdout: 10 189994 23.16 MB/sec execute 305 sec latency 103.205 ms 2024-05-08T20:28:41.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 350925 43.94 MB/sec execute 305 sec latency 608.449 ms 2024-05-08T20:28:42.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:42 smithi188 ceph-mon[48761]: pgmap v878: 129 pgs: 129 active+clean; 432 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 500 op/s 2024-05-08T20:28:42.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:42 smithi083 ceph-mon[38695]: pgmap v878: 129 pgs: 129 active+clean; 432 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 500 op/s 2024-05-08T20:28:42.522 INFO:tasks.workunit.client.1.smithi188.stdout: 10 190402 23.17 MB/sec execute 306 sec latency 124.374 ms 2024-05-08T20:28:42.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 351896 43.97 MB/sec execute 306 sec latency 538.227 ms 2024-05-08T20:28:43.522 INFO:tasks.workunit.client.1.smithi188.stdout: 10 190915 23.16 MB/sec execute 307 sec latency 92.661 ms 2024-05-08T20:28:43.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 352095 43.84 MB/sec execute 307 sec latency 117.215 ms 2024-05-08T20:28:44.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:44 smithi188 ceph-mon[48761]: pgmap v879: 129 pgs: 129 active+clean; 433 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 8.9 MiB/s wr, 448 op/s 2024-05-08T20:28:44.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:44 smithi083 ceph-mon[38695]: pgmap v879: 129 pgs: 129 active+clean; 433 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 8.9 MiB/s wr, 448 op/s 2024-05-08T20:28:44.522 INFO:tasks.workunit.client.1.smithi188.stdout: 10 191278 23.14 MB/sec execute 308 sec latency 90.300 ms 2024-05-08T20:28:44.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 353572 43.90 MB/sec execute 308 sec latency 138.005 ms 2024-05-08T20:28:45.522 INFO:tasks.workunit.client.1.smithi188.stdout: 10 191846 23.21 MB/sec execute 309 sec latency 103.594 ms 2024-05-08T20:28:45.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 354934 44.07 MB/sec execute 309 sec latency 128.086 ms 2024-05-08T20:28:46.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:46 smithi188 ceph-mon[48761]: pgmap v880: 129 pgs: 129 active+clean; 432 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 514 op/s 2024-05-08T20:28:46.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:46 smithi083 ceph-mon[38695]: pgmap v880: 129 pgs: 129 active+clean; 432 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 514 op/s 2024-05-08T20:28:46.522 INFO:tasks.workunit.client.1.smithi188.stdout: 10 192237 23.21 MB/sec execute 310 sec latency 110.228 ms 2024-05-08T20:28:46.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 355479 44.02 MB/sec execute 310 sec latency 128.363 ms 2024-05-08T20:28:47.524 INFO:tasks.workunit.client.1.smithi188.stdout: 10 192663 23.21 MB/sec execute 311 sec latency 93.172 ms 2024-05-08T20:28:47.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 355784 43.92 MB/sec execute 311 sec latency 269.975 ms 2024-05-08T20:28:48.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:48 smithi188 ceph-mon[48761]: pgmap v881: 129 pgs: 129 active+clean; 431 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 20 KiB/s rd, 11 MiB/s wr, 548 op/s 2024-05-08T20:28:48.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:48 smithi083 ceph-mon[38695]: pgmap v881: 129 pgs: 129 active+clean; 431 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 20 KiB/s rd, 11 MiB/s wr, 548 op/s 2024-05-08T20:28:48.524 INFO:tasks.workunit.client.1.smithi188.stdout: 10 193079 23.19 MB/sec execute 312 sec latency 104.066 ms 2024-05-08T20:28:48.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 357202 43.98 MB/sec execute 312 sec latency 145.553 ms 2024-05-08T20:28:49.524 INFO:tasks.workunit.client.1.smithi188.stdout: 10 193430 23.16 MB/sec execute 313 sec latency 105.004 ms 2024-05-08T20:28:49.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 358485 44.12 MB/sec execute 313 sec latency 125.482 ms 2024-05-08T20:28:50.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:50 smithi188 ceph-mon[48761]: pgmap v882: 129 pgs: 129 active+clean; 429 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 474 op/s 2024-05-08T20:28:50.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:50 smithi083 ceph-mon[38695]: pgmap v882: 129 pgs: 129 active+clean; 429 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 474 op/s 2024-05-08T20:28:50.525 INFO:tasks.workunit.client.1.smithi188.stdout: 10 193853 23.18 MB/sec execute 314 sec latency 158.072 ms 2024-05-08T20:28:50.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 359051 44.07 MB/sec execute 314 sec latency 118.579 ms 2024-05-08T20:28:51.525 INFO:tasks.workunit.client.1.smithi188.stdout: 10 194285 23.15 MB/sec execute 315 sec latency 79.475 ms 2024-05-08T20:28:51.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 359827 44.08 MB/sec execute 315 sec latency 111.888 ms 2024-05-08T20:28:52.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:52 smithi083 ceph-mon[38695]: pgmap v883: 129 pgs: 129 active+clean; 437 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 577 op/s 2024-05-08T20:28:52.525 INFO:tasks.workunit.client.1.smithi188.stdout: 10 194723 23.15 MB/sec execute 316 sec latency 95.847 ms 2024-05-08T20:28:52.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:52 smithi188 ceph-mon[48761]: pgmap v883: 129 pgs: 129 active+clean; 437 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 577 op/s 2024-05-08T20:28:52.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 361074 44.14 MB/sec execute 316 sec latency 243.181 ms 2024-05-08T20:28:53.525 INFO:tasks.workunit.client.1.smithi188.stdout: 10 195148 23.18 MB/sec execute 317 sec latency 127.665 ms 2024-05-08T20:28:53.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 362476 44.25 MB/sec execute 317 sec latency 110.659 ms 2024-05-08T20:28:54.404 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:54 smithi083 ceph-mon[38695]: pgmap v884: 129 pgs: 129 active+clean; 435 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 14 KiB/s rd, 8.9 MiB/s wr, 435 op/s 2024-05-08T20:28:54.404 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:28:54.528 INFO:tasks.workunit.client.1.smithi188.stdout: 10 195566 23.17 MB/sec execute 318 sec latency 94.783 ms 2024-05-08T20:28:54.598 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:54 smithi188 ceph-mon[48761]: pgmap v884: 129 pgs: 129 active+clean; 435 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 14 KiB/s rd, 8.9 MiB/s wr, 435 op/s 2024-05-08T20:28:54.598 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:28:54.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 362598 44.12 MB/sec execute 318 sec latency 125.327 ms 2024-05-08T20:28:55.528 INFO:tasks.workunit.client.1.smithi188.stdout: 10 195935 23.19 MB/sec execute 319 sec latency 105.177 ms 2024-05-08T20:28:55.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 363556 44.13 MB/sec execute 319 sec latency 123.887 ms 2024-05-08T20:28:56.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:56 smithi083 ceph-mon[38695]: pgmap v885: 129 pgs: 129 active+clean; 424 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 475 op/s 2024-05-08T20:28:56.528 INFO:tasks.workunit.client.1.smithi188.stdout: 10 196216 23.16 MB/sec execute 320 sec latency 136.446 ms 2024-05-08T20:28:56.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:56 smithi188 ceph-mon[48761]: pgmap v885: 129 pgs: 129 active+clean; 424 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 475 op/s 2024-05-08T20:28:56.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 364821 44.22 MB/sec execute 320 sec latency 122.562 ms 2024-05-08T20:28:57.528 INFO:tasks.workunit.client.1.smithi188.stdout: 10 196632 23.14 MB/sec execute 321 sec latency 31.106 ms 2024-05-08T20:28:57.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 366024 44.29 MB/sec execute 321 sec latency 238.944 ms 2024-05-08T20:28:58.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:28:58 smithi083 ceph-mon[38695]: pgmap v886: 129 pgs: 129 active+clean; 431 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 20 KiB/s rd, 11 MiB/s wr, 546 op/s 2024-05-08T20:28:58.529 INFO:tasks.workunit.client.1.smithi188.stdout: 10 197015 23.12 MB/sec execute 322 sec latency 129.338 ms 2024-05-08T20:28:58.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:28:58 smithi188 ceph-mon[48761]: pgmap v886: 129 pgs: 129 active+clean; 431 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 20 KiB/s rd, 11 MiB/s wr, 546 op/s 2024-05-08T20:28:58.775 INFO:tasks.workunit.client.0.smithi083.stdout: 10 366146 44.16 MB/sec execute 322 sec latency 337.129 ms 2024-05-08T20:28:59.528 INFO:tasks.workunit.client.1.smithi188.stdout: 10 197438 23.13 MB/sec execute 323 sec latency 130.032 ms 2024-05-08T20:28:59.777 INFO:tasks.workunit.client.0.smithi083.stdout: 10 367029 44.17 MB/sec execute 323 sec latency 127.253 ms 2024-05-08T20:29:00.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:00 smithi083 ceph-mon[38695]: pgmap v887: 129 pgs: 129 active+clean; 439 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 486 op/s 2024-05-08T20:29:00.529 INFO:tasks.workunit.client.1.smithi188.stdout: 10 197827 23.09 MB/sec execute 324 sec latency 112.151 ms 2024-05-08T20:29:00.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:00 smithi188 ceph-mon[48761]: pgmap v887: 129 pgs: 129 active+clean; 439 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 486 op/s 2024-05-08T20:29:00.777 INFO:tasks.workunit.client.0.smithi083.stdout: 10 368369 44.27 MB/sec execute 324 sec latency 119.916 ms 2024-05-08T20:29:01.529 INFO:tasks.workunit.client.1.smithi188.stdout: 10 198262 23.10 MB/sec execute 325 sec latency 110.074 ms 2024-05-08T20:29:01.777 INFO:tasks.workunit.client.0.smithi083.stdout: 10 369579 44.33 MB/sec execute 325 sec latency 132.601 ms 2024-05-08T20:29:02.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:02 smithi083 ceph-mon[38695]: pgmap v888: 129 pgs: 129 active+clean; 435 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 20 KiB/s rd, 12 MiB/s wr, 588 op/s 2024-05-08T20:29:02.529 INFO:tasks.workunit.client.1.smithi188.stdout: 10 198734 23.13 MB/sec execute 326 sec latency 90.870 ms 2024-05-08T20:29:02.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:02 smithi188 ceph-mon[48761]: pgmap v888: 129 pgs: 129 active+clean; 435 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 20 KiB/s rd, 12 MiB/s wr, 588 op/s 2024-05-08T20:29:02.777 INFO:tasks.workunit.client.0.smithi083.stdout: 10 369709 44.21 MB/sec execute 326 sec latency 120.445 ms 2024-05-08T20:29:03.529 INFO:tasks.workunit.client.1.smithi188.stdout: 10 199182 23.13 MB/sec execute 327 sec latency 113.425 ms 2024-05-08T20:29:03.777 INFO:tasks.workunit.client.0.smithi083.stdout: 10 370648 44.22 MB/sec execute 327 sec latency 217.600 ms 2024-05-08T20:29:04.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:04 smithi083 ceph-mon[38695]: pgmap v889: 129 pgs: 129 active+clean; 432 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 8.6 MiB/s wr, 432 op/s 2024-05-08T20:29:04.529 INFO:tasks.workunit.client.1.smithi188.stdout: 10 199559 23.15 MB/sec execute 328 sec latency 95.248 ms 2024-05-08T20:29:04.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:04 smithi188 ceph-mon[48761]: pgmap v889: 129 pgs: 129 active+clean; 432 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 8.6 MiB/s wr, 432 op/s 2024-05-08T20:29:04.777 INFO:tasks.workunit.client.0.smithi083.stdout: 10 371913 44.31 MB/sec execute 328 sec latency 138.310 ms 2024-05-08T20:29:05.529 INFO:tasks.workunit.client.1.smithi188.stdout: 10 199854 23.12 MB/sec execute 329 sec latency 143.356 ms 2024-05-08T20:29:05.777 INFO:tasks.workunit.client.0.smithi083.stdout: 10 373161 44.38 MB/sec execute 329 sec latency 120.135 ms 2024-05-08T20:29:06.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:06 smithi083 ceph-mon[38695]: pgmap v890: 129 pgs: 129 active+clean; 426 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 464 op/s 2024-05-08T20:29:06.529 INFO:tasks.workunit.client.1.smithi188.stdout: 10 200250 23.09 MB/sec execute 330 sec latency 40.207 ms 2024-05-08T20:29:06.599 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:06 smithi188 ceph-mon[48761]: pgmap v890: 129 pgs: 129 active+clean; 426 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 464 op/s 2024-05-08T20:29:06.777 INFO:tasks.workunit.client.0.smithi083.stdout: 10 373324 44.26 MB/sec execute 330 sec latency 150.892 ms 2024-05-08T20:29:07.536 INFO:tasks.workunit.client.1.smithi188.stdout: 10 200630 23.08 MB/sec execute 331 sec latency 116.593 ms 2024-05-08T20:29:07.777 INFO:tasks.workunit.client.0.smithi083.stdout: 10 374756 44.31 MB/sec execute 331 sec latency 118.982 ms 2024-05-08T20:29:08.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:08 smithi083 ceph-mon[38695]: pgmap v891: 129 pgs: 129 active+clean; 435 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 547 op/s 2024-05-08T20:29:08.535 INFO:tasks.workunit.client.1.smithi188.stdout: 10 200979 23.08 MB/sec execute 332 sec latency 143.217 ms 2024-05-08T20:29:08.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:08 smithi188 ceph-mon[48761]: pgmap v891: 129 pgs: 129 active+clean; 435 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 547 op/s 2024-05-08T20:29:08.777 INFO:tasks.workunit.client.0.smithi083.stdout: 10 375747 44.40 MB/sec execute 332 sec latency 164.920 ms 2024-05-08T20:29:09.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:29:09.535 INFO:tasks.workunit.client.1.smithi188.stdout: 10 201427 23.06 MB/sec execute 333 sec latency 33.794 ms 2024-05-08T20:29:09.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:29:09.777 INFO:tasks.workunit.client.0.smithi083.stdout: 10 376715 44.42 MB/sec execute 333 sec latency 118.070 ms 2024-05-08T20:29:10.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:10 smithi083 ceph-mon[38695]: pgmap v892: 129 pgs: 129 active+clean; 440 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 489 op/s 2024-05-08T20:29:10.541 INFO:tasks.workunit.client.1.smithi188.stdout: 10 201854 23.05 MB/sec execute 334 sec latency 113.351 ms 2024-05-08T20:29:10.598 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:10 smithi188 ceph-mon[48761]: pgmap v892: 129 pgs: 129 active+clean; 440 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 489 op/s 2024-05-08T20:29:10.777 INFO:tasks.workunit.client.0.smithi083.stdout: 10 376932 44.30 MB/sec execute 334 sec latency 125.504 ms 2024-05-08T20:29:11.542 INFO:tasks.workunit.client.1.smithi188.stdout: 10 202294 23.08 MB/sec execute 335 sec latency 116.906 ms 2024-05-08T20:29:11.777 INFO:tasks.workunit.client.0.smithi083.stdout: 10 378499 44.38 MB/sec execute 335 sec latency 118.055 ms 2024-05-08T20:29:12.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:12 smithi083 ceph-mon[38695]: pgmap v893: 129 pgs: 129 active+clean; 420 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 10 MiB/s wr, 490 op/s 2024-05-08T20:29:12.541 INFO:tasks.workunit.client.1.smithi188.stdout: 10 202782 23.09 MB/sec execute 336 sec latency 100.023 ms 2024-05-08T20:29:12.601 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:12 smithi188 ceph-mon[48761]: pgmap v893: 129 pgs: 129 active+clean; 420 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 10 MiB/s wr, 490 op/s 2024-05-08T20:29:12.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 380240 44.59 MB/sec execute 336 sec latency 112.601 ms 2024-05-08T20:29:13.541 INFO:tasks.workunit.client.1.smithi188.stdout: 10 203243 23.12 MB/sec execute 337 sec latency 70.811 ms 2024-05-08T20:29:13.777 INFO:tasks.workunit.client.0.smithi083.stdout: 10 380303 44.46 MB/sec execute 337 sec latency 339.183 ms 2024-05-08T20:29:14.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:14 smithi083 ceph-mon[38695]: pgmap v894: 129 pgs: 129 active+clean; 423 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 368 op/s 2024-05-08T20:29:14.542 INFO:tasks.workunit.client.1.smithi188.stdout: 10 203537 23.09 MB/sec execute 338 sec latency 104.171 ms 2024-05-08T20:29:14.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:14 smithi188 ceph-mon[48761]: pgmap v894: 129 pgs: 129 active+clean; 423 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 368 op/s 2024-05-08T20:29:14.777 INFO:tasks.workunit.client.0.smithi083.stdout: 10 380317 44.33 MB/sec execute 338 sec latency 683.068 ms 2024-05-08T20:29:15.544 INFO:tasks.workunit.client.1.smithi188.stdout: 10 203820 23.04 MB/sec execute 339 sec latency 118.782 ms 2024-05-08T20:29:15.777 INFO:tasks.workunit.client.0.smithi083.stdout: 10 380359 44.20 MB/sec execute 339 sec latency 761.463 ms 2024-05-08T20:29:16.544 INFO:tasks.workunit.client.1.smithi188.stdout: 10 204198 23.03 MB/sec execute 340 sec latency 65.252 ms 2024-05-08T20:29:16.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:16 smithi188 ceph-mon[48761]: pgmap v895: 129 pgs: 129 active+clean; 429 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 465 op/s 2024-05-08T20:29:16.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:16 smithi083 ceph-mon[38695]: pgmap v895: 129 pgs: 129 active+clean; 429 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 465 op/s 2024-05-08T20:29:16.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 380369 44.07 MB/sec execute 340 sec latency 555.686 ms 2024-05-08T20:29:17.544 INFO:tasks.workunit.client.1.smithi188.stdout: 10 204627 23.04 MB/sec execute 341 sec latency 120.102 ms 2024-05-08T20:29:17.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 380388 43.95 MB/sec execute 341 sec latency 529.308 ms 2024-05-08T20:29:18.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:18 smithi083 ceph-mon[38695]: pgmap v896: 129 pgs: 129 active+clean; 436 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 520 op/s 2024-05-08T20:29:18.544 INFO:tasks.workunit.client.1.smithi188.stdout: 10 205030 23.02 MB/sec execute 342 sec latency 130.961 ms 2024-05-08T20:29:18.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:18 smithi188 ceph-mon[48761]: pgmap v896: 129 pgs: 129 active+clean; 436 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 520 op/s 2024-05-08T20:29:18.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 381547 43.97 MB/sec execute 342 sec latency 356.978 ms 2024-05-08T20:29:19.544 INFO:tasks.workunit.client.1.smithi188.stdout: 10 205453 23.01 MB/sec execute 343 sec latency 101.191 ms 2024-05-08T20:29:19.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 382854 44.09 MB/sec execute 343 sec latency 131.182 ms 2024-05-08T20:29:20.349 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:20 smithi188 ceph-mon[48761]: pgmap v897: 129 pgs: 129 active+clean; 437 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 14 KiB/s rd, 9.6 MiB/s wr, 458 op/s 2024-05-08T20:29:20.545 INFO:tasks.workunit.client.1.smithi188.stdout: 10 205878 23.03 MB/sec execute 344 sec latency 107.670 ms 2024-05-08T20:29:20.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:20 smithi083 ceph-mon[38695]: pgmap v897: 129 pgs: 129 active+clean; 437 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 14 KiB/s rd, 9.6 MiB/s wr, 458 op/s 2024-05-08T20:29:20.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 383840 44.11 MB/sec execute 344 sec latency 197.690 ms 2024-05-08T20:29:21.545 INFO:tasks.workunit.client.1.smithi188.stdout: 10 206344 23.04 MB/sec execute 345 sec latency 80.824 ms 2024-05-08T20:29:21.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 384038 44.00 MB/sec execute 345 sec latency 109.390 ms 2024-05-08T20:29:22.546 INFO:tasks.workunit.client.1.smithi188.stdout: 10 206764 23.06 MB/sec execute 346 sec latency 102.063 ms 2024-05-08T20:29:22.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:22 smithi188 ceph-mon[48761]: pgmap v898: 129 pgs: 129 active+clean; 431 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 10 MiB/s wr, 477 op/s 2024-05-08T20:29:22.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:22 smithi083 ceph-mon[38695]: pgmap v898: 129 pgs: 129 active+clean; 431 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 19 KiB/s rd, 10 MiB/s wr, 477 op/s 2024-05-08T20:29:22.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 385542 44.06 MB/sec execute 346 sec latency 119.828 ms 2024-05-08T20:29:23.546 INFO:tasks.workunit.client.1.smithi188.stdout: 10 207150 23.05 MB/sec execute 347 sec latency 134.046 ms 2024-05-08T20:29:23.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 386447 44.14 MB/sec execute 347 sec latency 223.241 ms 2024-05-08T20:29:24.546 INFO:tasks.workunit.client.1.smithi188.stdout: 10 207444 23.02 MB/sec execute 348 sec latency 121.070 ms 2024-05-08T20:29:24.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:24 smithi188 ceph-mon[48761]: pgmap v899: 129 pgs: 129 active+clean; 429 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 8.6 MiB/s wr, 414 op/s 2024-05-08T20:29:24.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:24 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:29:24.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:24 smithi083 ceph-mon[38695]: pgmap v899: 129 pgs: 129 active+clean; 429 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 8.6 MiB/s wr, 414 op/s 2024-05-08T20:29:24.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:29:24.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 387417 44.16 MB/sec execute 348 sec latency 128.027 ms 2024-05-08T20:29:25.546 INFO:tasks.workunit.client.1.smithi188.stdout: 10 207776 22.99 MB/sec execute 349 sec latency 129.167 ms 2024-05-08T20:29:25.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 387641 44.05 MB/sec execute 349 sec latency 131.882 ms 2024-05-08T20:29:26.546 INFO:tasks.workunit.client.1.smithi188.stdout: 10 208083 22.98 MB/sec execute 350 sec latency 90.317 ms 2024-05-08T20:29:26.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:26 smithi188 ceph-mon[48761]: pgmap v900: 129 pgs: 129 active+clean; 430 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 544 op/s 2024-05-08T20:29:26.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:26 smithi083 ceph-mon[38695]: pgmap v900: 129 pgs: 129 active+clean; 430 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 544 op/s 2024-05-08T20:29:26.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 389177 44.13 MB/sec execute 350 sec latency 132.079 ms 2024-05-08T20:29:27.546 INFO:tasks.workunit.client.1.smithi188.stdout: 10 208437 22.96 MB/sec execute 351 sec latency 71.414 ms 2024-05-08T20:29:27.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 390809 44.31 MB/sec execute 351 sec latency 132.997 ms 2024-05-08T20:29:28.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:28 smithi083 ceph-mon[38695]: pgmap v901: 129 pgs: 129 active+clean; 436 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 18 KiB/s rd, 10 MiB/s wr, 479 op/s 2024-05-08T20:29:28.546 INFO:tasks.workunit.client.1.smithi188.stdout: 10 208840 22.94 MB/sec execute 352 sec latency 125.276 ms 2024-05-08T20:29:28.598 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:28 smithi188 ceph-mon[48761]: pgmap v901: 129 pgs: 129 active+clean; 436 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 18 KiB/s rd, 10 MiB/s wr, 479 op/s 2024-05-08T20:29:28.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 390958 44.20 MB/sec execute 352 sec latency 369.326 ms 2024-05-08T20:29:29.546 INFO:tasks.workunit.client.1.smithi188.stdout: 10 209250 22.95 MB/sec execute 353 sec latency 94.730 ms 2024-05-08T20:29:29.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 391157 44.09 MB/sec execute 353 sec latency 97.935 ms 2024-05-08T20:29:30.549 INFO:tasks.workunit.client.1.smithi188.stdout: 10 209742 22.98 MB/sec execute 354 sec latency 84.568 ms 2024-05-08T20:29:30.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:30 smithi188 ceph-mon[48761]: pgmap v902: 129 pgs: 129 active+clean; 431 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 11 KiB/s rd, 9.1 MiB/s wr, 486 op/s 2024-05-08T20:29:30.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:30 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:29:30.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:30 smithi083 ceph-mon[38695]: pgmap v902: 129 pgs: 129 active+clean; 431 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 11 KiB/s rd, 9.1 MiB/s wr, 486 op/s 2024-05-08T20:29:30.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:30 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:29:30.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 392633 44.14 MB/sec execute 354 sec latency 126.093 ms 2024-05-08T20:29:31.548 INFO:tasks.workunit.client.1.smithi188.stdout: 10 210101 22.98 MB/sec execute 355 sec latency 117.482 ms 2024-05-08T20:29:31.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:31 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:29:31.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:31 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:29:31.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:31 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:29:31.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:31 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:29:31.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:31 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:29:31.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:31 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:29:31.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 394444 44.36 MB/sec execute 355 sec latency 123.343 ms 2024-05-08T20:29:32.549 INFO:tasks.workunit.client.1.smithi188.stdout: 10 210471 22.97 MB/sec execute 356 sec latency 89.444 ms 2024-05-08T20:29:32.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:32 smithi188 ceph-mon[48761]: pgmap v903: 129 pgs: 129 active+clean; 420 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 18 KiB/s rd, 10 MiB/s wr, 484 op/s 2024-05-08T20:29:32.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:32 smithi083 ceph-mon[38695]: pgmap v903: 129 pgs: 129 active+clean; 420 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 18 KiB/s rd, 10 MiB/s wr, 484 op/s 2024-05-08T20:29:32.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 394570 44.24 MB/sec execute 356 sec latency 112.632 ms 2024-05-08T20:29:33.549 INFO:tasks.workunit.client.1.smithi188.stdout: 10 210814 22.95 MB/sec execute 357 sec latency 133.967 ms 2024-05-08T20:29:33.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 395409 44.25 MB/sec execute 357 sec latency 126.359 ms 2024-05-08T20:29:34.552 INFO:tasks.workunit.client.1.smithi188.stdout: 10 211120 22.92 MB/sec execute 358 sec latency 117.514 ms 2024-05-08T20:29:34.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:34 smithi188 ceph-mon[48761]: pgmap v904: 129 pgs: 129 active+clean; 422 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 8.1 MiB/s wr, 389 op/s 2024-05-08T20:29:34.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:34 smithi083 ceph-mon[38695]: pgmap v904: 129 pgs: 129 active+clean; 422 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 8.1 MiB/s wr, 389 op/s 2024-05-08T20:29:34.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 396705 44.33 MB/sec execute 358 sec latency 157.164 ms 2024-05-08T20:29:35.549 INFO:tasks.workunit.client.1.smithi188.stdout: 10 211455 22.90 MB/sec execute 359 sec latency 136.023 ms 2024-05-08T20:29:35.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 398005 44.40 MB/sec execute 359 sec latency 131.013 ms 2024-05-08T20:29:36.549 INFO:tasks.workunit.client.1.smithi188.stdout: 10 211854 22.91 MB/sec execute 360 sec latency 128.337 ms 2024-05-08T20:29:36.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:36 smithi188 ceph-mon[48761]: pgmap v905: 129 pgs: 129 active+clean; 433 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 513 op/s 2024-05-08T20:29:36.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:36 smithi083 ceph-mon[38695]: pgmap v905: 129 pgs: 129 active+clean; 433 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 513 op/s 2024-05-08T20:29:36.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 398134 44.28 MB/sec execute 360 sec latency 110.631 ms 2024-05-08T20:29:37.549 INFO:tasks.workunit.client.1.smithi188.stdout: 10 212294 22.88 MB/sec execute 361 sec latency 107.820 ms 2024-05-08T20:29:37.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 399200 44.30 MB/sec execute 361 sec latency 137.402 ms 2024-05-08T20:29:38.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:38 smithi083 ceph-mon[38695]: pgmap v906: 129 pgs: 129 active+clean; 434 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 17 KiB/s rd, 9.6 MiB/s wr, 418 op/s 2024-05-08T20:29:38.549 INFO:tasks.workunit.client.1.smithi188.stdout: 10 212715 22.89 MB/sec execute 362 sec latency 103.780 ms 2024-05-08T20:29:38.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:38 smithi188 ceph-mon[48761]: pgmap v906: 129 pgs: 129 active+clean; 434 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 17 KiB/s rd, 9.6 MiB/s wr, 418 op/s 2024-05-08T20:29:38.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 400535 44.40 MB/sec execute 362 sec latency 125.234 ms 2024-05-08T20:29:39.549 INFO:tasks.workunit.client.1.smithi188.stdout: 10 213197 22.91 MB/sec execute 363 sec latency 70.064 ms 2024-05-08T20:29:39.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:29:39.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:29:39.778 INFO:tasks.workunit.client.0.smithi083.stdout: 10 401588 44.44 MB/sec execute 363 sec latency 230.441 ms 2024-05-08T20:29:40.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:40 smithi083 ceph-mon[38695]: pgmap v907: 129 pgs: 129 active+clean; 430 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 395 op/s 2024-05-08T20:29:40.549 INFO:tasks.workunit.client.1.smithi188.stdout: 10 213531 22.90 MB/sec execute 364 sec latency 129.023 ms 2024-05-08T20:29:40.600 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:40 smithi188 ceph-mon[48761]: pgmap v907: 129 pgs: 129 active+clean; 430 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 395 op/s 2024-05-08T20:29:40.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 401751 44.33 MB/sec execute 364 sec latency 107.684 ms 2024-05-08T20:29:41.549 INFO:tasks.workunit.client.1.smithi188.stdout: 10 214059 22.94 MB/sec execute 365 sec latency 37.879 ms 2024-05-08T20:29:41.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 403183 44.37 MB/sec execute 365 sec latency 111.666 ms 2024-05-08T20:29:42.549 INFO:tasks.workunit.client.1.smithi188.stdout: 10 214464 22.92 MB/sec execute 366 sec latency 72.650 ms 2024-05-08T20:29:42.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:42 smithi188 ceph-mon[48761]: pgmap v908: 129 pgs: 129 active+clean; 427 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 18 KiB/s rd, 10 MiB/s wr, 474 op/s 2024-05-08T20:29:42.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:42 smithi083 ceph-mon[38695]: pgmap v908: 129 pgs: 129 active+clean; 427 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 18 KiB/s rd, 10 MiB/s wr, 474 op/s 2024-05-08T20:29:42.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 404176 44.46 MB/sec execute 366 sec latency 134.047 ms 2024-05-08T20:29:43.549 INFO:tasks.workunit.client.1.smithi188.stdout: 10 214801 22.90 MB/sec execute 367 sec latency 120.398 ms 2024-05-08T20:29:43.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 405154 44.48 MB/sec execute 367 sec latency 112.661 ms 2024-05-08T20:29:44.550 INFO:tasks.workunit.client.1.smithi188.stdout: 10 215094 22.87 MB/sec execute 368 sec latency 137.397 ms 2024-05-08T20:29:44.598 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:44 smithi188 ceph-mon[48761]: pgmap v909: 129 pgs: 129 active+clean; 426 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 8.6 MiB/s wr, 412 op/s 2024-05-08T20:29:44.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:44 smithi083 ceph-mon[38695]: pgmap v909: 129 pgs: 129 active+clean; 426 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 8.6 MiB/s wr, 412 op/s 2024-05-08T20:29:44.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 405484 44.39 MB/sec execute 368 sec latency 150.974 ms 2024-05-08T20:29:45.550 INFO:tasks.workunit.client.1.smithi188.stdout: 10 215430 22.87 MB/sec execute 369 sec latency 167.416 ms 2024-05-08T20:29:45.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 407047 44.49 MB/sec execute 369 sec latency 123.777 ms 2024-05-08T20:29:46.550 INFO:tasks.workunit.client.1.smithi188.stdout: 10 215826 22.84 MB/sec execute 370 sec latency 106.699 ms 2024-05-08T20:29:46.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:46 smithi188 ceph-mon[48761]: pgmap v910: 129 pgs: 129 active+clean; 434 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 505 op/s 2024-05-08T20:29:46.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:46 smithi083 ceph-mon[38695]: pgmap v910: 129 pgs: 129 active+clean; 434 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 505 op/s 2024-05-08T20:29:46.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 408628 44.63 MB/sec execute 370 sec latency 128.880 ms 2024-05-08T20:29:47.550 INFO:tasks.workunit.client.1.smithi188.stdout: 10 216191 22.83 MB/sec execute 371 sec latency 118.641 ms 2024-05-08T20:29:47.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 408755 44.52 MB/sec execute 371 sec latency 109.290 ms 2024-05-08T20:29:48.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:48 smithi083 ceph-mon[38695]: pgmap v911: 129 pgs: 129 active+clean; 431 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 18 KiB/s rd, 10 MiB/s wr, 465 op/s 2024-05-08T20:29:48.550 INFO:tasks.workunit.client.1.smithi188.stdout: 10 216659 22.86 MB/sec execute 372 sec latency 100.096 ms 2024-05-08T20:29:48.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:48 smithi188 ceph-mon[48761]: pgmap v911: 129 pgs: 129 active+clean; 431 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 18 KiB/s rd, 10 MiB/s wr, 465 op/s 2024-05-08T20:29:48.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 409714 44.53 MB/sec execute 372 sec latency 121.384 ms 2024-05-08T20:29:49.552 INFO:tasks.workunit.client.1.smithi188.stdout: 10 217122 22.86 MB/sec execute 373 sec latency 81.772 ms 2024-05-08T20:29:49.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 410460 44.48 MB/sec execute 373 sec latency 353.651 ms 2024-05-08T20:29:50.552 INFO:tasks.workunit.client.1.smithi188.stdout: 10 217506 22.88 MB/sec execute 374 sec latency 112.077 ms 2024-05-08T20:29:50.578 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:50 smithi188 ceph-mon[48761]: pgmap v912: 129 pgs: 129 active+clean; 428 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 8.8 MiB/s wr, 440 op/s 2024-05-08T20:29:50.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:50 smithi083 ceph-mon[38695]: pgmap v912: 129 pgs: 129 active+clean; 428 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 8.8 MiB/s wr, 440 op/s 2024-05-08T20:29:50.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 410886 44.48 MB/sec execute 374 sec latency 413.345 ms 2024-05-08T20:29:51.552 INFO:tasks.workunit.client.1.smithi188.stdout: 10 217867 22.86 MB/sec execute 375 sec latency 89.323 ms 2024-05-08T20:29:51.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:51 smithi188 ceph-mon[48761]: pgmap v913: 129 pgs: 129 active+clean; 432 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 608 op/s 2024-05-08T20:29:51.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:51 smithi083 ceph-mon[38695]: pgmap v913: 129 pgs: 129 active+clean; 432 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 608 op/s 2024-05-08T20:29:51.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 410998 44.37 MB/sec execute 375 sec latency 608.088 ms 2024-05-08T20:29:52.552 INFO:tasks.workunit.client.1.smithi188.stdout: 10 218202 22.84 MB/sec execute 376 sec latency 86.058 ms 2024-05-08T20:29:52.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 411208 44.28 MB/sec execute 376 sec latency 586.781 ms 2024-05-08T20:29:53.552 INFO:tasks.workunit.client.1.smithi188.stdout: 10 218560 22.82 MB/sec execute 377 sec latency 108.145 ms 2024-05-08T20:29:53.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 412174 44.31 MB/sec execute 377 sec latency 430.967 ms 2024-05-08T20:29:53.849 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:53 smithi188 ceph-mon[48761]: pgmap v914: 129 pgs: 129 active+clean; 425 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 432 op/s 2024-05-08T20:29:53.948 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:53 smithi083 ceph-mon[38695]: pgmap v914: 129 pgs: 129 active+clean; 425 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 432 op/s 2024-05-08T20:29:54.552 INFO:tasks.workunit.client.1.smithi188.stdout: 10 218857 22.81 MB/sec execute 378 sec latency 124.141 ms 2024-05-08T20:29:54.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 412300 44.20 MB/sec execute 378 sec latency 110.118 ms 2024-05-08T20:29:54.846 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:29:54.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:29:55.552 INFO:tasks.workunit.client.1.smithi188.stdout: 10 219331 22.80 MB/sec execute 379 sec latency 34.049 ms 2024-05-08T20:29:55.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:55 smithi083 ceph-mon[38695]: pgmap v915: 129 pgs: 129 active+clean; 428 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 462 op/s 2024-05-08T20:29:55.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 413256 44.21 MB/sec execute 379 sec latency 113.574 ms 2024-05-08T20:29:55.848 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:55 smithi188 ceph-mon[48761]: pgmap v915: 129 pgs: 129 active+clean; 428 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 462 op/s 2024-05-08T20:29:56.552 INFO:tasks.workunit.client.1.smithi188.stdout: 10 219706 22.79 MB/sec execute 380 sec latency 123.342 ms 2024-05-08T20:29:56.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 414545 44.29 MB/sec execute 380 sec latency 153.966 ms 2024-05-08T20:29:57.552 INFO:tasks.workunit.client.1.smithi188.stdout: 10 220198 22.81 MB/sec execute 381 sec latency 33.554 ms 2024-05-08T20:29:57.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 415751 44.35 MB/sec execute 381 sec latency 125.885 ms 2024-05-08T20:29:57.846 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:57 smithi188 ceph-mon[48761]: pgmap v916: 129 pgs: 129 active+clean; 424 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 20 KiB/s rd, 10 MiB/s wr, 495 op/s 2024-05-08T20:29:57.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:57 smithi083 ceph-mon[38695]: pgmap v916: 129 pgs: 129 active+clean; 424 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 20 KiB/s rd, 10 MiB/s wr, 495 op/s 2024-05-08T20:29:58.552 INFO:tasks.workunit.client.1.smithi188.stdout: 10 220609 22.81 MB/sec execute 382 sec latency 81.623 ms 2024-05-08T20:29:58.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 415888 44.24 MB/sec execute 382 sec latency 203.376 ms 2024-05-08T20:29:59.552 INFO:tasks.workunit.client.1.smithi188.stdout: 10 220933 22.82 MB/sec execute 383 sec latency 87.921 ms 2024-05-08T20:29:59.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 417022 44.26 MB/sec execute 383 sec latency 144.979 ms 2024-05-08T20:29:59.847 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:29:59 smithi188 ceph-mon[48761]: pgmap v917: 129 pgs: 129 active+clean; 422 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 8.5 MiB/s wr, 426 op/s 2024-05-08T20:29:59.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:29:59 smithi083 ceph-mon[38695]: pgmap v917: 129 pgs: 129 active+clean; 422 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 8.5 MiB/s wr, 426 op/s 2024-05-08T20:30:00.552 INFO:tasks.workunit.client.1.smithi188.stdout: 10 221277 22.81 MB/sec execute 384 sec latency 96.229 ms 2024-05-08T20:30:00.747 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:00 smithi188 ceph-mon[48761]: overall HEALTH_OK 2024-05-08T20:30:00.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 418266 44.35 MB/sec execute 384 sec latency 146.866 ms 2024-05-08T20:30:00.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:00 smithi083 ceph-mon[38695]: overall HEALTH_OK 2024-05-08T20:30:01.552 INFO:tasks.workunit.client.1.smithi188.stdout: 10 221616 22.78 MB/sec execute 385 sec latency 107.638 ms 2024-05-08T20:30:01.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 419313 44.38 MB/sec execute 385 sec latency 156.450 ms 2024-05-08T20:30:01.847 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:01 smithi188 ceph-mon[48761]: pgmap v918: 129 pgs: 129 active+clean; 429 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 517 op/s 2024-05-08T20:30:01.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:01 smithi083 ceph-mon[38695]: pgmap v918: 129 pgs: 129 active+clean; 429 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 517 op/s 2024-05-08T20:30:02.552 INFO:tasks.workunit.client.1.smithi188.stdout: 10 221991 22.77 MB/sec execute 386 sec latency 94.548 ms 2024-05-08T20:30:02.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 419488 44.28 MB/sec execute 386 sec latency 109.030 ms 2024-05-08T20:30:03.552 INFO:tasks.workunit.client.1.smithi188.stdout: 10 222372 22.78 MB/sec execute 387 sec latency 121.404 ms 2024-05-08T20:30:03.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:03 smithi083 ceph-mon[38695]: pgmap v919: 129 pgs: 129 active+clean; 434 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 386 op/s 2024-05-08T20:30:03.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 420565 44.29 MB/sec execute 387 sec latency 201.118 ms 2024-05-08T20:30:03.846 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:03 smithi188 ceph-mon[48761]: pgmap v919: 129 pgs: 129 active+clean; 434 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 386 op/s 2024-05-08T20:30:04.552 INFO:tasks.workunit.client.1.smithi188.stdout: 10 222705 22.76 MB/sec execute 388 sec latency 142.661 ms 2024-05-08T20:30:04.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 421604 44.36 MB/sec execute 388 sec latency 306.805 ms 2024-05-08T20:30:05.552 INFO:tasks.workunit.client.1.smithi188.stdout: 10 223123 22.74 MB/sec execute 389 sec latency 105.631 ms 2024-05-08T20:30:05.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 422827 44.42 MB/sec execute 389 sec latency 118.410 ms 2024-05-08T20:30:05.827 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:05 smithi188 ceph-mon[48761]: pgmap v920: 129 pgs: 129 active+clean; 439 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 458 op/s 2024-05-08T20:30:05.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:05 smithi083 ceph-mon[38695]: pgmap v920: 129 pgs: 129 active+clean; 439 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 458 op/s 2024-05-08T20:30:06.552 INFO:tasks.workunit.client.1.smithi188.stdout: 10 223487 22.73 MB/sec execute 390 sec latency 84.293 ms 2024-05-08T20:30:06.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 422940 44.31 MB/sec execute 390 sec latency 129.070 ms 2024-05-08T20:30:07.553 INFO:tasks.workunit.client.1.smithi188.stdout: 10 224019 22.76 MB/sec execute 391 sec latency 90.491 ms 2024-05-08T20:30:07.779 INFO:tasks.workunit.client.0.smithi083.stdout: 10 423897 44.32 MB/sec execute 391 sec latency 149.657 ms 2024-05-08T20:30:07.846 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:07 smithi188 ceph-mon[48761]: pgmap v921: 129 pgs: 129 active+clean; 431 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 19 KiB/s rd, 10 MiB/s wr, 515 op/s 2024-05-08T20:30:07.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:07 smithi083 ceph-mon[38695]: pgmap v921: 129 pgs: 129 active+clean; 431 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 19 KiB/s rd, 10 MiB/s wr, 515 op/s 2024-05-08T20:30:08.553 INFO:tasks.workunit.client.1.smithi188.stdout: 10 224436 22.77 MB/sec execute 392 sec latency 116.796 ms 2024-05-08T20:30:08.780 INFO:tasks.workunit.client.0.smithi083.stdout: 10 425153 44.39 MB/sec execute 392 sec latency 144.660 ms 2024-05-08T20:30:09.555 INFO:tasks.workunit.client.1.smithi188.stdout: 10 224844 22.78 MB/sec execute 393 sec latency 71.548 ms 2024-05-08T20:30:09.780 INFO:tasks.workunit.client.0.smithi083.stdout: 10 426358 44.45 MB/sec execute 393 sec latency 191.752 ms 2024-05-08T20:30:09.846 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:09 smithi188 ceph-mon[48761]: pgmap v922: 129 pgs: 129 active+clean; 431 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 386 op/s 2024-05-08T20:30:09.847 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:30:09.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:09 smithi083 ceph-mon[38695]: pgmap v922: 129 pgs: 129 active+clean; 431 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 386 op/s 2024-05-08T20:30:09.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:30:10.555 INFO:tasks.workunit.client.1.smithi188.stdout: 10 225231 22.77 MB/sec execute 394 sec latency 134.042 ms 2024-05-08T20:30:10.780 INFO:tasks.workunit.client.0.smithi083.stdout: 10 426482 44.34 MB/sec execute 394 sec latency 134.541 ms 2024-05-08T20:30:11.555 INFO:tasks.workunit.client.1.smithi188.stdout: 10 225573 22.73 MB/sec execute 395 sec latency 116.501 ms 2024-05-08T20:30:11.780 INFO:tasks.workunit.client.0.smithi083.stdout: 10 427438 44.35 MB/sec execute 395 sec latency 134.872 ms 2024-05-08T20:30:11.846 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:11 smithi188 ceph-mon[48761]: pgmap v923: 129 pgs: 129 active+clean; 437 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 16 KiB/s rd, 12 MiB/s wr, 564 op/s 2024-05-08T20:30:11.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:11 smithi083 ceph-mon[38695]: pgmap v923: 129 pgs: 129 active+clean; 437 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 16 KiB/s rd, 12 MiB/s wr, 564 op/s 2024-05-08T20:30:12.555 INFO:tasks.workunit.client.1.smithi188.stdout: 10 225964 22.75 MB/sec execute 396 sec latency 157.503 ms 2024-05-08T20:30:12.780 INFO:tasks.workunit.client.0.smithi083.stdout: 10 428696 44.43 MB/sec execute 396 sec latency 140.082 ms 2024-05-08T20:30:13.555 INFO:tasks.workunit.client.1.smithi188.stdout: 10 226346 22.73 MB/sec execute 397 sec latency 67.804 ms 2024-05-08T20:30:13.780 INFO:tasks.workunit.client.0.smithi083.stdout: 10 429918 44.48 MB/sec execute 397 sec latency 139.841 ms 2024-05-08T20:30:13.846 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:13 smithi188 ceph-mon[48761]: pgmap v924: 129 pgs: 129 active+clean; 437 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 461 op/s 2024-05-08T20:30:13.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:13 smithi083 ceph-mon[38695]: pgmap v924: 129 pgs: 129 active+clean; 437 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 461 op/s 2024-05-08T20:30:14.555 INFO:tasks.workunit.client.1.smithi188.stdout: 10 226772 22.71 MB/sec execute 398 sec latency 105.738 ms 2024-05-08T20:30:14.780 INFO:tasks.workunit.client.0.smithi083.stdout: 10 430029 44.38 MB/sec execute 398 sec latency 228.709 ms 2024-05-08T20:30:15.556 INFO:tasks.workunit.client.1.smithi188.stdout: 10 227218 22.72 MB/sec execute 399 sec latency 108.910 ms 2024-05-08T20:30:15.780 INFO:tasks.workunit.client.0.smithi083.stdout: 10 430850 44.38 MB/sec execute 399 sec latency 132.423 ms 2024-05-08T20:30:15.846 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:15 smithi188 ceph-mon[48761]: pgmap v925: 129 pgs: 129 active+clean; 428 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 9.8 MiB/s wr, 442 op/s 2024-05-08T20:30:15.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:15 smithi083 ceph-mon[38695]: pgmap v925: 129 pgs: 129 active+clean; 428 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 9.8 MiB/s wr, 442 op/s 2024-05-08T20:30:16.555 INFO:tasks.workunit.client.1.smithi188.stdout: 10 227732 22.75 MB/sec execute 400 sec latency 53.438 ms 2024-05-08T20:30:16.780 INFO:tasks.workunit.client.0.smithi083.stdout: 10 432159 44.45 MB/sec execute 400 sec latency 123.089 ms 2024-05-08T20:30:17.556 INFO:tasks.workunit.client.1.smithi188.stdout: 10 228129 22.77 MB/sec execute 401 sec latency 147.157 ms 2024-05-08T20:30:17.780 INFO:tasks.workunit.client.0.smithi083.stdout: 10 433449 44.52 MB/sec execute 401 sec latency 139.898 ms 2024-05-08T20:30:17.846 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:17 smithi188 ceph-mon[48761]: pgmap v926: 129 pgs: 129 active+clean; 426 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 19 KiB/s rd, 10 MiB/s wr, 518 op/s 2024-05-08T20:30:17.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:17 smithi083 ceph-mon[38695]: pgmap v926: 129 pgs: 129 active+clean; 426 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 19 KiB/s rd, 10 MiB/s wr, 518 op/s 2024-05-08T20:30:18.556 INFO:tasks.workunit.client.1.smithi188.stdout: 10 228494 22.75 MB/sec execute 402 sec latency 112.986 ms 2024-05-08T20:30:18.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 433570 44.41 MB/sec execute 402 sec latency 148.400 ms 2024-05-08T20:30:19.556 INFO:tasks.workunit.client.1.smithi188.stdout: 10 228855 22.74 MB/sec execute 403 sec latency 138.393 ms 2024-05-08T20:30:19.780 INFO:tasks.workunit.client.0.smithi083.stdout: 10 434279 44.41 MB/sec execute 403 sec latency 137.129 ms 2024-05-08T20:30:19.847 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:19 smithi188 ceph-mon[48761]: pgmap v927: 129 pgs: 129 active+clean; 430 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 9.4 MiB/s wr, 445 op/s 2024-05-08T20:30:19.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:19 smithi083 ceph-mon[38695]: pgmap v927: 129 pgs: 129 active+clean; 430 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 9.4 MiB/s wr, 445 op/s 2024-05-08T20:30:20.556 INFO:tasks.workunit.client.1.smithi188.stdout: 10 229200 22.72 MB/sec execute 404 sec latency 113.433 ms 2024-05-08T20:30:20.780 INFO:tasks.workunit.client.0.smithi083.stdout: 10 435638 44.48 MB/sec execute 404 sec latency 137.632 ms 2024-05-08T20:30:21.556 INFO:tasks.workunit.client.1.smithi188.stdout: 10 229574 22.72 MB/sec execute 405 sec latency 159.814 ms 2024-05-08T20:30:21.781 INFO:tasks.workunit.client.0.smithi083.stdout: 10 436996 44.55 MB/sec execute 405 sec latency 122.825 ms 2024-05-08T20:30:21.846 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:21 smithi188 ceph-mon[48761]: pgmap v928: 129 pgs: 129 active+clean; 431 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 570 op/s 2024-05-08T20:30:21.948 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:21 smithi083 ceph-mon[38695]: pgmap v928: 129 pgs: 129 active+clean; 431 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 570 op/s 2024-05-08T20:30:22.559 INFO:tasks.workunit.client.1.smithi188.stdout: 10 230034 22.71 MB/sec execute 406 sec latency 84.016 ms 2024-05-08T20:30:22.780 INFO:tasks.workunit.client.0.smithi083.stdout: 10 437120 44.45 MB/sec execute 406 sec latency 118.238 ms 2024-05-08T20:30:23.559 INFO:tasks.workunit.client.1.smithi188.stdout: 10 230493 22.71 MB/sec execute 407 sec latency 134.339 ms 2024-05-08T20:30:23.780 INFO:tasks.workunit.client.0.smithi083.stdout: 10 438078 44.45 MB/sec execute 407 sec latency 138.521 ms 2024-05-08T20:30:23.847 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:23 smithi188 ceph-mon[48761]: pgmap v929: 129 pgs: 129 active+clean; 430 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 8.2 MiB/s wr, 377 op/s 2024-05-08T20:30:23.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:23 smithi083 ceph-mon[38695]: pgmap v929: 129 pgs: 129 active+clean; 430 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 8.2 MiB/s wr, 377 op/s 2024-05-08T20:30:24.559 INFO:tasks.workunit.client.1.smithi188.stdout: 10 230947 22.73 MB/sec execute 408 sec latency 66.268 ms 2024-05-08T20:30:24.780 INFO:tasks.workunit.client.0.smithi083.stdout: 10 439312 44.53 MB/sec execute 408 sec latency 190.494 ms 2024-05-08T20:30:24.847 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:24 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:30:24.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:30:25.559 INFO:tasks.workunit.client.1.smithi188.stdout: 10 231403 22.74 MB/sec execute 409 sec latency 112.221 ms 2024-05-08T20:30:25.780 INFO:tasks.workunit.client.0.smithi083.stdout: 10 439450 44.43 MB/sec execute 409 sec latency 559.464 ms 2024-05-08T20:30:25.846 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:25 smithi188 ceph-mon[48761]: pgmap v930: 129 pgs: 129 active+clean; 428 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 9.7 MiB/s wr, 456 op/s 2024-05-08T20:30:25.856 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:25 smithi083 ceph-mon[38695]: pgmap v930: 129 pgs: 129 active+clean; 428 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 9.7 MiB/s wr, 456 op/s 2024-05-08T20:30:26.559 INFO:tasks.workunit.client.1.smithi188.stdout: 10 231750 22.74 MB/sec execute 410 sec latency 85.771 ms 2024-05-08T20:30:26.780 INFO:tasks.workunit.client.0.smithi083.stdout: 10 439605 44.35 MB/sec execute 410 sec latency 758.897 ms 2024-05-08T20:30:27.559 INFO:tasks.workunit.client.1.smithi188.stdout: 10 232061 22.73 MB/sec execute 411 sec latency 110.696 ms 2024-05-08T20:30:27.780 INFO:tasks.workunit.client.0.smithi083.stdout: 10 439808 44.27 MB/sec execute 411 sec latency 633.011 ms 2024-05-08T20:30:27.847 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:27 smithi188 ceph-mon[48761]: pgmap v931: 129 pgs: 129 active+clean; 436 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 20 KiB/s rd, 11 MiB/s wr, 570 op/s 2024-05-08T20:30:27.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:27 smithi083 ceph-mon[38695]: pgmap v931: 129 pgs: 129 active+clean; 436 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 20 KiB/s rd, 11 MiB/s wr, 570 op/s 2024-05-08T20:30:28.560 INFO:tasks.workunit.client.1.smithi188.stdout: 10 232419 22.71 MB/sec execute 412 sec latency 91.049 ms 2024-05-08T20:30:28.780 INFO:tasks.workunit.client.0.smithi083.stdout: 10 440179 44.21 MB/sec execute 412 sec latency 509.002 ms 2024-05-08T20:30:29.564 INFO:tasks.workunit.client.1.smithi188.stdout: 10 232764 22.68 MB/sec execute 413 sec latency 91.720 ms 2024-05-08T20:30:29.780 INFO:tasks.workunit.client.0.smithi083.stdout: 10 440595 44.15 MB/sec execute 413 sec latency 438.729 ms 2024-05-08T20:30:29.846 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:29 smithi188 ceph-mon[48761]: pgmap v932: 129 pgs: 129 active+clean; 439 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 430 op/s 2024-05-08T20:30:29.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:29 smithi083 ceph-mon[38695]: pgmap v932: 129 pgs: 129 active+clean; 439 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 430 op/s 2024-05-08T20:30:30.566 INFO:tasks.workunit.client.1.smithi188.stdout: 10 233137 22.68 MB/sec execute 414 sec latency 157.126 ms 2024-05-08T20:30:30.781 INFO:tasks.workunit.client.0.smithi083.stdout: 10 440813 44.06 MB/sec execute 414 sec latency 159.383 ms 2024-05-08T20:30:31.566 INFO:tasks.workunit.client.1.smithi188.stdout: 10 233530 22.67 MB/sec execute 415 sec latency 118.241 ms 2024-05-08T20:30:31.781 INFO:tasks.workunit.client.0.smithi083.stdout: 10 442357 44.13 MB/sec execute 415 sec latency 130.241 ms 2024-05-08T20:30:31.847 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:31 smithi188 ceph-mon[48761]: pgmap v933: 129 pgs: 129 active+clean; 442 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 542 op/s 2024-05-08T20:30:31.847 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:31 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:30:31.847 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:31 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:30:31.847 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:31 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:30:31.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:31 smithi083 ceph-mon[38695]: pgmap v933: 129 pgs: 129 active+clean; 442 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 542 op/s 2024-05-08T20:30:31.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:31 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:30:31.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:31 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:30:31.947 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:31 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:30:32.566 INFO:tasks.workunit.client.1.smithi188.stdout: 10 233979 22.67 MB/sec execute 416 sec latency 99.443 ms 2024-05-08T20:30:32.781 INFO:tasks.workunit.client.0.smithi083.stdout: 10 443635 44.23 MB/sec execute 416 sec latency 136.718 ms 2024-05-08T20:30:32.847 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:32 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:30:32.847 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:32 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:30:32.847 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:32 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:30:32.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:32 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:30:32.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:32 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:30:32.946 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:32 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:30:33.566 INFO:tasks.workunit.client.1.smithi188.stdout: 10 234409 22.67 MB/sec execute 417 sec latency 64.646 ms 2024-05-08T20:30:33.781 INFO:tasks.workunit.client.0.smithi083.stdout: 10 444177 44.19 MB/sec execute 417 sec latency 134.516 ms 2024-05-08T20:30:34.196 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:33 smithi083 ceph-mon[38695]: pgmap v934: 129 pgs: 129 active+clean; 441 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 8.7 MiB/s wr, 404 op/s 2024-05-08T20:30:34.348 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:33 smithi188 ceph-mon[48761]: pgmap v934: 129 pgs: 129 active+clean; 441 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 8.7 MiB/s wr, 404 op/s 2024-05-08T20:30:34.566 INFO:tasks.workunit.client.1.smithi188.stdout: 10 234863 22.69 MB/sec execute 418 sec latency 115.422 ms 2024-05-08T20:30:34.781 INFO:tasks.workunit.client.0.smithi083.stdout: 10 444309 44.09 MB/sec execute 418 sec latency 388.063 ms 2024-05-08T20:30:35.566 INFO:tasks.workunit.client.1.smithi188.stdout: 10 235269 22.71 MB/sec execute 419 sec latency 100.341 ms 2024-05-08T20:30:35.781 INFO:tasks.workunit.client.0.smithi083.stdout: 10 445394 44.10 MB/sec execute 419 sec latency 309.731 ms 2024-05-08T20:30:36.196 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:35 smithi083 ceph-mon[38695]: pgmap v935: 129 pgs: 129 active+clean; 440 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 476 op/s 2024-05-08T20:30:36.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:35 smithi188 ceph-mon[48761]: pgmap v935: 129 pgs: 129 active+clean; 440 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 476 op/s 2024-05-08T20:30:36.566 INFO:tasks.workunit.client.1.smithi188.stdout: 10 235631 22.70 MB/sec execute 420 sec latency 111.955 ms 2024-05-08T20:30:36.781 INFO:tasks.workunit.client.0.smithi083.stdout: 10 446711 44.21 MB/sec execute 420 sec latency 121.560 ms 2024-05-08T20:30:37.566 INFO:tasks.workunit.client.1.smithi188.stdout: 10 235993 22.68 MB/sec execute 421 sec latency 83.072 ms 2024-05-08T20:30:37.781 INFO:tasks.workunit.client.0.smithi083.stdout: 10 447683 44.22 MB/sec execute 421 sec latency 127.489 ms 2024-05-08T20:30:38.196 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:37 smithi083 ceph-mon[38695]: pgmap v936: 129 pgs: 129 active+clean; 441 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 530 op/s 2024-05-08T20:30:38.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:37 smithi188 ceph-mon[48761]: pgmap v936: 129 pgs: 129 active+clean; 441 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 530 op/s 2024-05-08T20:30:38.566 INFO:tasks.workunit.client.1.smithi188.stdout: 10 236343 22.66 MB/sec execute 422 sec latency 118.556 ms 2024-05-08T20:30:38.781 INFO:tasks.workunit.client.0.smithi083.stdout: 10 447851 44.13 MB/sec execute 422 sec latency 146.164 ms 2024-05-08T20:30:39.196 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:38 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:30:39.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:38 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:30:39.566 INFO:tasks.workunit.client.1.smithi188.stdout: 10 236697 22.65 MB/sec execute 423 sec latency 131.713 ms 2024-05-08T20:30:39.781 INFO:tasks.workunit.client.0.smithi083.stdout: 10 449126 44.15 MB/sec execute 423 sec latency 129.737 ms 2024-05-08T20:30:40.197 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:39 smithi083 ceph-mon[38695]: pgmap v937: 129 pgs: 129 active+clean; 439 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 396 op/s 2024-05-08T20:30:40.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:39 smithi188 ceph-mon[48761]: pgmap v937: 129 pgs: 129 active+clean; 439 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 396 op/s 2024-05-08T20:30:40.567 INFO:tasks.workunit.client.1.smithi188.stdout: 10 237134 22.66 MB/sec execute 424 sec latency 86.291 ms 2024-05-08T20:30:40.781 INFO:tasks.workunit.client.0.smithi083.stdout: 10 450262 44.24 MB/sec execute 424 sec latency 133.384 ms 2024-05-08T20:30:41.568 INFO:tasks.workunit.client.1.smithi188.stdout: 10 237562 22.65 MB/sec execute 425 sec latency 80.316 ms 2024-05-08T20:30:41.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 451235 44.25 MB/sec execute 425 sec latency 108.916 ms 2024-05-08T20:30:42.196 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:41 smithi083 ceph-mon[38695]: pgmap v938: 129 pgs: 129 active+clean; 436 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 548 op/s 2024-05-08T20:30:42.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:41 smithi188 ceph-mon[48761]: pgmap v938: 129 pgs: 129 active+clean; 436 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 548 op/s 2024-05-08T20:30:42.568 INFO:tasks.workunit.client.1.smithi188.stdout: 10 237974 22.64 MB/sec execute 426 sec latency 94.107 ms 2024-05-08T20:30:42.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 451453 44.16 MB/sec execute 426 sec latency 115.432 ms 2024-05-08T20:30:43.568 INFO:tasks.workunit.client.1.smithi188.stdout: 10 238387 22.66 MB/sec execute 427 sec latency 100.965 ms 2024-05-08T20:30:43.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 452901 44.21 MB/sec execute 427 sec latency 137.940 ms 2024-05-08T20:30:44.196 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:43 smithi083 ceph-mon[38695]: pgmap v939: 129 pgs: 129 active+clean; 433 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 8.5 MiB/s wr, 431 op/s 2024-05-08T20:30:44.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:43 smithi188 ceph-mon[48761]: pgmap v939: 129 pgs: 129 active+clean; 433 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 8.5 MiB/s wr, 431 op/s 2024-05-08T20:30:44.568 INFO:tasks.workunit.client.1.smithi188.stdout: 10 238729 22.65 MB/sec execute 428 sec latency 104.037 ms 2024-05-08T20:30:44.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 454003 44.30 MB/sec execute 428 sec latency 143.941 ms 2024-05-08T20:30:45.568 INFO:tasks.workunit.client.1.smithi188.stdout: 10 239140 22.66 MB/sec execute 429 sec latency 96.406 ms 2024-05-08T20:30:45.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 454792 44.29 MB/sec execute 429 sec latency 177.338 ms 2024-05-08T20:30:46.196 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:45 smithi083 ceph-mon[38695]: pgmap v940: 129 pgs: 129 active+clean; 431 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 485 op/s 2024-05-08T20:30:46.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:45 smithi188 ceph-mon[48761]: pgmap v940: 129 pgs: 129 active+clean; 431 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 485 op/s 2024-05-08T20:30:46.568 INFO:tasks.workunit.client.1.smithi188.stdout: 10 239533 22.65 MB/sec execute 430 sec latency 112.686 ms 2024-05-08T20:30:46.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 455121 44.22 MB/sec execute 430 sec latency 126.990 ms 2024-05-08T20:30:47.569 INFO:tasks.workunit.client.1.smithi188.stdout: 10 239925 22.64 MB/sec execute 431 sec latency 99.180 ms 2024-05-08T20:30:47.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 456580 44.27 MB/sec execute 431 sec latency 130.597 ms 2024-05-08T20:30:48.196 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:47 smithi083 ceph-mon[38695]: pgmap v941: 129 pgs: 129 active+clean; 432 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 19 KiB/s rd, 10 MiB/s wr, 496 op/s 2024-05-08T20:30:48.349 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:47 smithi188 ceph-mon[48761]: pgmap v941: 129 pgs: 129 active+clean; 432 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 19 KiB/s rd, 10 MiB/s wr, 496 op/s 2024-05-08T20:30:48.569 INFO:tasks.workunit.client.1.smithi188.stdout: 10 240304 22.63 MB/sec execute 432 sec latency 126.317 ms 2024-05-08T20:30:48.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 458174 44.41 MB/sec execute 432 sec latency 134.293 ms 2024-05-08T20:30:49.569 INFO:tasks.workunit.client.1.smithi188.stdout: 10 240709 22.63 MB/sec execute 433 sec latency 100.353 ms 2024-05-08T20:30:49.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 459062 44.81 MB/sec execute 433 sec latency 136.574 ms 2024-05-08T20:30:50.198 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:49 smithi083 ceph-mon[38695]: pgmap v942: 129 pgs: 129 active+clean; 437 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 9.1 MiB/s wr, 424 op/s 2024-05-08T20:30:50.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:49 smithi188 ceph-mon[48761]: pgmap v942: 129 pgs: 129 active+clean; 437 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 9.1 MiB/s wr, 424 op/s 2024-05-08T20:30:50.569 INFO:tasks.workunit.client.1.smithi188.stdout: 10 241157 22.62 MB/sec execute 434 sec latency 87.558 ms 2024-05-08T20:30:50.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 459226 44.71 MB/sec execute 434 sec latency 125.728 ms 2024-05-08T20:30:51.569 INFO:tasks.workunit.client.1.smithi188.stdout: 10 241613 22.62 MB/sec execute 435 sec latency 93.388 ms 2024-05-08T20:30:51.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 460512 44.74 MB/sec execute 435 sec latency 118.290 ms 2024-05-08T20:30:52.198 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:51 smithi083 ceph-mon[38695]: pgmap v943: 129 pgs: 129 active+clean; 426 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 586 op/s 2024-05-08T20:30:52.349 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:51 smithi188 ceph-mon[48761]: pgmap v943: 129 pgs: 129 active+clean; 426 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 586 op/s 2024-05-08T20:30:52.569 INFO:tasks.workunit.client.1.smithi188.stdout: 10 242110 22.65 MB/sec execute 436 sec latency 105.167 ms 2024-05-08T20:30:52.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 461639 44.82 MB/sec execute 436 sec latency 152.437 ms 2024-05-08T20:30:53.569 INFO:tasks.workunit.client.1.smithi188.stdout: 10 242527 22.66 MB/sec execute 437 sec latency 89.358 ms 2024-05-08T20:30:53.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 462642 44.84 MB/sec execute 437 sec latency 129.413 ms 2024-05-08T20:30:54.197 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:53 smithi083 ceph-mon[38695]: pgmap v944: 129 pgs: 129 active+clean; 426 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 14 KiB/s rd, 8.8 MiB/s wr, 426 op/s 2024-05-08T20:30:54.197 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:53 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:30:54.350 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:53 smithi188 ceph-mon[48761]: pgmap v944: 129 pgs: 129 active+clean; 426 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 14 KiB/s rd, 8.8 MiB/s wr, 426 op/s 2024-05-08T20:30:54.350 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:53 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:30:54.569 INFO:tasks.workunit.client.1.smithi188.stdout: 10 242834 22.64 MB/sec execute 438 sec latency 115.650 ms 2024-05-08T20:30:54.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 462812 44.74 MB/sec execute 438 sec latency 116.052 ms 2024-05-08T20:30:55.569 INFO:tasks.workunit.client.1.smithi188.stdout: 10 243217 22.63 MB/sec execute 439 sec latency 98.954 ms 2024-05-08T20:30:55.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 464227 44.78 MB/sec execute 439 sec latency 142.744 ms 2024-05-08T20:30:56.198 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:55 smithi083 ceph-mon[38695]: pgmap v945: 129 pgs: 129 active+clean; 424 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 457 op/s 2024-05-08T20:30:56.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:55 smithi188 ceph-mon[48761]: pgmap v945: 129 pgs: 129 active+clean; 424 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 457 op/s 2024-05-08T20:30:56.569 INFO:tasks.workunit.client.1.smithi188.stdout: 10 243578 22.62 MB/sec execute 440 sec latency 84.641 ms 2024-05-08T20:30:56.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 465220 44.85 MB/sec execute 440 sec latency 125.087 ms 2024-05-08T20:30:57.569 INFO:tasks.workunit.client.1.smithi188.stdout: 10 243935 22.61 MB/sec execute 441 sec latency 78.236 ms 2024-05-08T20:30:57.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 466186 44.86 MB/sec execute 441 sec latency 113.914 ms 2024-05-08T20:30:58.196 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:57 smithi083 ceph-mon[38695]: pgmap v946: 129 pgs: 129 active+clean; 422 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 20 KiB/s rd, 11 MiB/s wr, 516 op/s 2024-05-08T20:30:58.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:57 smithi188 ceph-mon[48761]: pgmap v946: 129 pgs: 129 active+clean; 422 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 20 KiB/s rd, 11 MiB/s wr, 516 op/s 2024-05-08T20:30:58.569 INFO:tasks.workunit.client.1.smithi188.stdout: 10 244344 22.61 MB/sec execute 442 sec latency 140.746 ms 2024-05-08T20:30:58.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 466383 44.77 MB/sec execute 442 sec latency 144.035 ms 2024-05-08T20:30:59.570 INFO:tasks.workunit.client.1.smithi188.stdout: 10 244800 22.60 MB/sec execute 443 sec latency 91.053 ms 2024-05-08T20:30:59.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 467853 44.82 MB/sec execute 443 sec latency 150.408 ms 2024-05-08T20:31:00.196 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:30:59 smithi083 ceph-mon[38695]: pgmap v947: 129 pgs: 129 active+clean; 422 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 444 op/s 2024-05-08T20:31:00.346 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:30:59 smithi188 ceph-mon[48761]: pgmap v947: 129 pgs: 129 active+clean; 422 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 444 op/s 2024-05-08T20:31:00.570 INFO:tasks.workunit.client.1.smithi188.stdout: 10 245272 22.62 MB/sec execute 444 sec latency 82.301 ms 2024-05-08T20:31:00.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 468710 44.87 MB/sec execute 444 sec latency 298.149 ms 2024-05-08T20:31:01.569 INFO:tasks.workunit.client.1.smithi188.stdout: 10 245709 22.64 MB/sec execute 445 sec latency 102.419 ms 2024-05-08T20:31:01.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 468923 44.80 MB/sec execute 445 sec latency 563.942 ms 2024-05-08T20:31:02.096 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:01 smithi188 ceph-mon[48761]: pgmap v948: 129 pgs: 129 active+clean; 416 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 20 KiB/s rd, 12 MiB/s wr, 600 op/s 2024-05-08T20:31:02.196 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:01 smithi083 ceph-mon[38695]: pgmap v948: 129 pgs: 129 active+clean; 416 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 20 KiB/s rd, 12 MiB/s wr, 600 op/s 2024-05-08T20:31:02.570 INFO:tasks.workunit.client.1.smithi188.stdout: 10 246082 22.64 MB/sec execute 446 sec latency 73.466 ms 2024-05-08T20:31:02.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 469224 44.73 MB/sec execute 446 sec latency 652.754 ms 2024-05-08T20:31:03.570 INFO:tasks.workunit.client.1.smithi188.stdout: 10 246462 22.63 MB/sec execute 447 sec latency 111.286 ms 2024-05-08T20:31:03.782 INFO:tasks.workunit.client.0.smithi083.stdout: 10 469672 44.69 MB/sec execute 447 sec latency 699.287 ms 2024-05-08T20:31:04.348 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:03 smithi188 ceph-mon[48761]: pgmap v949: 129 pgs: 129 active+clean; 415 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 431 op/s 2024-05-08T20:31:04.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:03 smithi083 ceph-mon[38695]: pgmap v949: 129 pgs: 129 active+clean; 415 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 431 op/s 2024-05-08T20:31:04.570 INFO:tasks.workunit.client.1.smithi188.stdout: 10 246819 22.61 MB/sec execute 448 sec latency 114.266 ms 2024-05-08T20:31:04.783 INFO:tasks.workunit.client.0.smithi083.stdout: 10 469677 44.59 MB/sec execute 448 sec latency 566.131 ms 2024-05-08T20:31:05.570 INFO:tasks.workunit.client.1.smithi188.stdout: 10 247185 22.60 MB/sec execute 449 sec latency 180.607 ms 2024-05-08T20:31:05.783 INFO:tasks.workunit.client.0.smithi083.stdout: 10 469746 44.49 MB/sec execute 449 sec latency 526.957 ms 2024-05-08T20:31:06.351 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:05 smithi188 ceph-mon[48761]: pgmap v950: 129 pgs: 129 active+clean; 422 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 458 op/s 2024-05-08T20:31:06.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:05 smithi083 ceph-mon[38695]: pgmap v950: 129 pgs: 129 active+clean; 422 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 458 op/s 2024-05-08T20:31:06.570 INFO:tasks.workunit.client.1.smithi188.stdout: 10 247546 22.59 MB/sec execute 450 sec latency 140.308 ms 2024-05-08T20:31:06.783 INFO:tasks.workunit.client.0.smithi083.stdout: 10 470262 44.45 MB/sec execute 450 sec latency 145.248 ms 2024-05-08T20:31:07.570 INFO:tasks.workunit.client.1.smithi188.stdout: 10 247960 22.59 MB/sec execute 451 sec latency 103.075 ms 2024-05-08T20:31:07.783 INFO:tasks.workunit.client.0.smithi083.stdout: 10 471643 44.50 MB/sec execute 451 sec latency 146.697 ms 2024-05-08T20:31:08.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:07 smithi188 ceph-mon[48761]: pgmap v951: 129 pgs: 129 active+clean; 430 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 560 op/s 2024-05-08T20:31:08.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:07 smithi083 ceph-mon[38695]: pgmap v951: 129 pgs: 129 active+clean; 430 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 560 op/s 2024-05-08T20:31:08.570 INFO:tasks.workunit.client.1.smithi188.stdout: 10 248341 22.57 MB/sec execute 452 sec latency 128.966 ms 2024-05-08T20:31:08.783 INFO:tasks.workunit.client.0.smithi083.stdout: 10 473219 44.61 MB/sec execute 452 sec latency 127.561 ms 2024-05-08T20:31:09.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:08 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:31:09.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:08 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:31:09.571 INFO:tasks.workunit.client.1.smithi188.stdout: 10 248744 22.57 MB/sec execute 453 sec latency 113.428 ms 2024-05-08T20:31:09.783 INFO:tasks.workunit.client.0.smithi083.stdout: 10 473341 44.52 MB/sec execute 453 sec latency 117.078 ms 2024-05-08T20:31:10.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:09 smithi188 ceph-mon[48761]: pgmap v952: 129 pgs: 129 active+clean; 427 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 9.2 MiB/s wr, 458 op/s 2024-05-08T20:31:10.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:09 smithi083 ceph-mon[38695]: pgmap v952: 129 pgs: 129 active+clean; 427 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 9.2 MiB/s wr, 458 op/s 2024-05-08T20:31:10.570 INFO:tasks.workunit.client.1.smithi188.stdout: 10 249211 22.60 MB/sec execute 454 sec latency 71.613 ms 2024-05-08T20:31:10.783 INFO:tasks.workunit.client.0.smithi083.stdout: 10 474052 44.52 MB/sec execute 454 sec latency 123.416 ms 2024-05-08T20:31:11.570 INFO:tasks.workunit.client.1.smithi188.stdout: 10 249635 22.61 MB/sec execute 455 sec latency 79.355 ms 2024-05-08T20:31:11.783 INFO:tasks.workunit.client.0.smithi083.stdout: 10 475476 44.59 MB/sec execute 455 sec latency 131.907 ms 2024-05-08T20:31:12.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:11 smithi188 ceph-mon[48761]: pgmap v953: 129 pgs: 129 active+clean; 426 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 607 op/s 2024-05-08T20:31:12.448 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:11 smithi083 ceph-mon[38695]: pgmap v953: 129 pgs: 129 active+clean; 426 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 607 op/s 2024-05-08T20:31:12.570 INFO:tasks.workunit.client.1.smithi188.stdout: 10 250028 22.61 MB/sec execute 456 sec latency 84.669 ms 2024-05-08T20:31:12.783 INFO:tasks.workunit.client.0.smithi083.stdout: 10 476772 44.64 MB/sec execute 456 sec latency 132.731 ms 2024-05-08T20:31:13.570 INFO:tasks.workunit.client.1.smithi188.stdout: 10 250406 22.59 MB/sec execute 457 sec latency 115.077 ms 2024-05-08T20:31:13.783 INFO:tasks.workunit.client.0.smithi083.stdout: 10 476912 44.55 MB/sec execute 457 sec latency 132.055 ms 2024-05-08T20:31:14.350 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:13 smithi188 ceph-mon[48761]: pgmap v954: 129 pgs: 129 active+clean; 427 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 390 op/s 2024-05-08T20:31:14.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:13 smithi083 ceph-mon[38695]: pgmap v954: 129 pgs: 129 active+clean; 427 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 390 op/s 2024-05-08T20:31:14.570 INFO:tasks.workunit.client.1.smithi188.stdout: 10 250689 22.57 MB/sec execute 458 sec latency 98.219 ms 2024-05-08T20:31:14.783 INFO:tasks.workunit.client.0.smithi083.stdout: 10 477955 44.56 MB/sec execute 458 sec latency 125.984 ms 2024-05-08T20:31:15.570 INFO:tasks.workunit.client.1.smithi188.stdout: 10 250982 22.56 MB/sec execute 459 sec latency 126.476 ms 2024-05-08T20:31:15.783 INFO:tasks.workunit.client.0.smithi083.stdout: 10 479240 44.63 MB/sec execute 459 sec latency 156.986 ms 2024-05-08T20:31:16.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:15 smithi188 ceph-mon[48761]: pgmap v955: 129 pgs: 129 active+clean; 425 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 429 op/s 2024-05-08T20:31:16.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:15 smithi083 ceph-mon[38695]: pgmap v955: 129 pgs: 129 active+clean; 425 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 429 op/s 2024-05-08T20:31:16.570 INFO:tasks.workunit.client.1.smithi188.stdout: 10 251307 22.54 MB/sec execute 460 sec latency 132.593 ms 2024-05-08T20:31:16.783 INFO:tasks.workunit.client.0.smithi083.stdout: 10 480379 44.67 MB/sec execute 460 sec latency 120.982 ms 2024-05-08T20:31:17.571 INFO:tasks.workunit.client.1.smithi188.stdout: 10 251682 22.54 MB/sec execute 461 sec latency 75.719 ms 2024-05-08T20:31:17.784 INFO:tasks.workunit.client.0.smithi083.stdout: 10 480516 44.58 MB/sec execute 461 sec latency 132.221 ms 2024-05-08T20:31:18.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:18 smithi188 ceph-mon[48761]: pgmap v956: 129 pgs: 129 active+clean; 428 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 19 KiB/s rd, 10 MiB/s wr, 532 op/s 2024-05-08T20:31:18.448 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:18 smithi083 ceph-mon[38695]: pgmap v956: 129 pgs: 129 active+clean; 428 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 19 KiB/s rd, 10 MiB/s wr, 532 op/s 2024-05-08T20:31:18.570 INFO:tasks.workunit.client.1.smithi188.stdout: 10 252171 22.54 MB/sec execute 462 sec latency 84.598 ms 2024-05-08T20:31:18.784 INFO:tasks.workunit.client.0.smithi083.stdout: 10 481651 44.60 MB/sec execute 462 sec latency 136.905 ms 2024-05-08T20:31:19.570 INFO:tasks.workunit.client.1.smithi188.stdout: 10 252578 22.55 MB/sec execute 463 sec latency 87.096 ms 2024-05-08T20:31:19.783 INFO:tasks.workunit.client.0.smithi083.stdout: 10 482985 44.69 MB/sec execute 463 sec latency 143.156 ms 2024-05-08T20:31:20.348 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:20 smithi188 ceph-mon[48761]: pgmap v957: 129 pgs: 129 active+clean; 425 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 8.8 MiB/s wr, 452 op/s 2024-05-08T20:31:20.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:20 smithi083 ceph-mon[38695]: pgmap v957: 129 pgs: 129 active+clean; 425 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 8.8 MiB/s wr, 452 op/s 2024-05-08T20:31:20.570 INFO:tasks.workunit.client.1.smithi188.stdout: 10 253009 22.57 MB/sec execute 464 sec latency 110.910 ms 2024-05-08T20:31:20.784 INFO:tasks.workunit.client.0.smithi083.stdout: 10 483957 44.70 MB/sec execute 464 sec latency 122.735 ms 2024-05-08T20:31:21.571 INFO:tasks.workunit.client.1.smithi188.stdout: 10 253268 22.54 MB/sec execute 465 sec latency 99.915 ms 2024-05-08T20:31:21.784 INFO:tasks.workunit.client.0.smithi083.stdout: 10 483997 44.61 MB/sec execute 465 sec latency 499.700 ms 2024-05-08T20:31:22.196 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:22 smithi083 ceph-mon[38695]: pgmap v958: 129 pgs: 129 active+clean; 425 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 550 op/s 2024-05-08T20:31:22.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:22 smithi188 ceph-mon[48761]: pgmap v958: 129 pgs: 129 active+clean; 425 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 550 op/s 2024-05-08T20:31:22.570 INFO:tasks.workunit.client.1.smithi188.stdout: 10 253635 22.53 MB/sec execute 466 sec latency 95.039 ms 2024-05-08T20:31:22.784 INFO:tasks.workunit.client.0.smithi083.stdout: 10 484023 44.51 MB/sec execute 466 sec latency 580.246 ms 2024-05-08T20:31:23.571 INFO:tasks.workunit.client.1.smithi188.stdout: 10 254018 22.52 MB/sec execute 467 sec latency 79.157 ms 2024-05-08T20:31:23.784 INFO:tasks.workunit.client.0.smithi083.stdout: 10 484066 44.42 MB/sec execute 467 sec latency 518.390 ms 2024-05-08T20:31:24.355 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:24 smithi188 ceph-mon[48761]: pgmap v959: 129 pgs: 129 active+clean; 425 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 8.5 MiB/s wr, 390 op/s 2024-05-08T20:31:24.355 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:24 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:31:24.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:24 smithi083 ceph-mon[38695]: pgmap v959: 129 pgs: 129 active+clean; 425 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 8.5 MiB/s wr, 390 op/s 2024-05-08T20:31:24.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:31:24.570 INFO:tasks.workunit.client.1.smithi188.stdout: 10 254400 22.52 MB/sec execute 468 sec latency 104.751 ms 2024-05-08T20:31:24.784 INFO:tasks.workunit.client.0.smithi083.stdout: 10 485202 44.43 MB/sec execute 468 sec latency 131.032 ms 2024-05-08T20:31:25.571 INFO:tasks.workunit.client.1.smithi188.stdout: 10 254785 22.50 MB/sec execute 469 sec latency 125.935 ms 2024-05-08T20:31:25.784 INFO:tasks.workunit.client.0.smithi083.stdout: 10 486473 44.52 MB/sec execute 469 sec latency 142.937 ms 2024-05-08T20:31:26.354 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:26 smithi188 ceph-mon[48761]: pgmap v960: 129 pgs: 129 active+clean; 430 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 475 op/s 2024-05-08T20:31:26.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:26 smithi083 ceph-mon[38695]: pgmap v960: 129 pgs: 129 active+clean; 430 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 475 op/s 2024-05-08T20:31:26.571 INFO:tasks.workunit.client.1.smithi188.stdout: 10 255167 22.51 MB/sec execute 470 sec latency 125.421 ms 2024-05-08T20:31:26.784 INFO:tasks.workunit.client.0.smithi083.stdout: 10 487499 44.54 MB/sec execute 470 sec latency 151.381 ms 2024-05-08T20:31:27.571 INFO:tasks.workunit.client.1.smithi188.stdout: 10 255675 22.51 MB/sec execute 471 sec latency 112.827 ms 2024-05-08T20:31:27.784 INFO:tasks.workunit.client.0.smithi083.stdout: 10 487694 44.45 MB/sec execute 471 sec latency 119.347 ms 2024-05-08T20:31:28.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:28 smithi188 ceph-mon[48761]: pgmap v961: 129 pgs: 129 active+clean; 431 MiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 19 KiB/s rd, 10 MiB/s wr, 514 op/s 2024-05-08T20:31:28.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:28 smithi083 ceph-mon[38695]: pgmap v961: 129 pgs: 129 active+clean; 431 MiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 19 KiB/s rd, 10 MiB/s wr, 514 op/s 2024-05-08T20:31:28.571 INFO:tasks.workunit.client.1.smithi188.stdout: 10 256076 22.51 MB/sec execute 472 sec latency 88.859 ms 2024-05-08T20:31:28.784 INFO:tasks.workunit.client.0.smithi083.stdout: 10 489134 44.49 MB/sec execute 472 sec latency 146.457 ms 2024-05-08T20:31:29.571 INFO:tasks.workunit.client.1.smithi188.stdout: 10 256497 22.52 MB/sec execute 473 sec latency 126.358 ms 2024-05-08T20:31:29.784 INFO:tasks.workunit.client.0.smithi083.stdout: 10 490080 44.55 MB/sec execute 473 sec latency 133.774 ms 2024-05-08T20:31:30.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:30 smithi188 ceph-mon[48761]: pgmap v962: 129 pgs: 129 active+clean; 433 MiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 11 KiB/s rd, 8.7 MiB/s wr, 450 op/s 2024-05-08T20:31:30.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:30 smithi083 ceph-mon[38695]: pgmap v962: 129 pgs: 129 active+clean; 433 MiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 11 KiB/s rd, 8.7 MiB/s wr, 450 op/s 2024-05-08T20:31:30.572 INFO:tasks.workunit.client.1.smithi188.stdout: 10 256889 22.53 MB/sec execute 474 sec latency 43.265 ms 2024-05-08T20:31:30.784 INFO:tasks.workunit.client.0.smithi083.stdout: 10 491063 44.57 MB/sec execute 474 sec latency 108.344 ms 2024-05-08T20:31:31.572 INFO:tasks.workunit.client.1.smithi188.stdout: 10 257240 22.51 MB/sec execute 475 sec latency 85.266 ms 2024-05-08T20:31:31.784 INFO:tasks.workunit.client.0.smithi083.stdout: 10 491398 44.50 MB/sec execute 475 sec latency 126.974 ms 2024-05-08T20:31:32.348 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:32 smithi188 ceph-mon[48761]: pgmap v963: 129 pgs: 129 active+clean; 430 MiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 514 op/s 2024-05-08T20:31:32.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:32 smithi083 ceph-mon[38695]: pgmap v963: 129 pgs: 129 active+clean; 430 MiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 514 op/s 2024-05-08T20:31:32.572 INFO:tasks.workunit.client.1.smithi188.stdout: 10 257584 22.50 MB/sec execute 476 sec latency 166.484 ms 2024-05-08T20:31:32.784 INFO:tasks.workunit.client.0.smithi083.stdout: 10 492990 44.58 MB/sec execute 476 sec latency 114.022 ms 2024-05-08T20:31:33.348 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:33 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:31:33.449 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:33 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:31:33.572 INFO:tasks.workunit.client.1.smithi188.stdout: 10 257919 22.49 MB/sec execute 477 sec latency 114.068 ms 2024-05-08T20:31:33.784 INFO:tasks.workunit.client.0.smithi083.stdout: 10 494531 44.68 MB/sec execute 477 sec latency 142.187 ms 2024-05-08T20:31:34.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:34 smithi188 ceph-mon[48761]: pgmap v964: 129 pgs: 129 active+clean; 427 MiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 400 op/s 2024-05-08T20:31:34.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:34 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:31:34.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:34 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:31:34.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:34 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:31:34.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:34 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:31:34.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:34 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:31:34.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:34 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:31:34.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:34 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:31:34.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:34 smithi083 ceph-mon[38695]: pgmap v964: 129 pgs: 129 active+clean; 427 MiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 400 op/s 2024-05-08T20:31:34.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:34 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:31:34.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:34 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:31:34.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:34 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:31:34.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:34 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:31:34.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:34 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:31:34.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:34 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:31:34.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:34 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:31:34.572 INFO:tasks.workunit.client.1.smithi188.stdout: 10 258248 22.47 MB/sec execute 478 sec latency 115.733 ms 2024-05-08T20:31:34.784 INFO:tasks.workunit.client.0.smithi083.stdout: 10 494562 44.59 MB/sec execute 478 sec latency 340.645 ms 2024-05-08T20:31:35.573 INFO:tasks.workunit.client.1.smithi188.stdout: 10 258660 22.47 MB/sec execute 479 sec latency 100.336 ms 2024-05-08T20:31:35.784 INFO:tasks.workunit.client.0.smithi083.stdout: 10 494757 44.51 MB/sec execute 479 sec latency 104.375 ms 2024-05-08T20:31:36.349 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:36 smithi188 ceph-mon[48761]: pgmap v965: 129 pgs: 129 active+clean; 422 MiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 475 op/s 2024-05-08T20:31:36.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:36 smithi083 ceph-mon[38695]: pgmap v965: 129 pgs: 129 active+clean; 422 MiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 475 op/s 2024-05-08T20:31:36.572 INFO:tasks.workunit.client.1.smithi188.stdout: 10 259078 22.47 MB/sec execute 480 sec latency 112.132 ms 2024-05-08T20:31:36.784 INFO:tasks.workunit.client.0.smithi083.stdout: 10 495978 44.53 MB/sec execute 480 sec latency 131.345 ms 2024-05-08T20:31:37.572 INFO:tasks.workunit.client.1.smithi188.stdout: 10 259490 22.47 MB/sec execute 481 sec latency 28.427 ms 2024-05-08T20:31:37.784 INFO:tasks.workunit.client.0.smithi083.stdout: 10 497163 44.61 MB/sec execute 481 sec latency 124.946 ms 2024-05-08T20:31:38.348 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:38 smithi188 ceph-mon[48761]: pgmap v966: 129 pgs: 129 active+clean; 426 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 19 KiB/s rd, 9.8 MiB/s wr, 494 op/s 2024-05-08T20:31:38.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:38 smithi083 ceph-mon[38695]: pgmap v966: 129 pgs: 129 active+clean; 426 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 19 KiB/s rd, 9.8 MiB/s wr, 494 op/s 2024-05-08T20:31:38.573 INFO:tasks.workunit.client.1.smithi188.stdout: 10 259942 22.50 MB/sec execute 482 sec latency 110.589 ms 2024-05-08T20:31:38.784 INFO:tasks.workunit.client.0.smithi083.stdout: 10 498138 44.62 MB/sec execute 482 sec latency 115.004 ms 2024-05-08T20:31:39.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:31:39.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:31:39.573 INFO:tasks.workunit.client.1.smithi188.stdout: 10 260320 22.48 MB/sec execute 483 sec latency 114.094 ms 2024-05-08T20:31:39.785 INFO:tasks.workunit.client.0.smithi083.stdout: 10 498356 44.54 MB/sec execute 483 sec latency 126.522 ms 2024-05-08T20:31:40.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:40 smithi188 ceph-mon[48761]: pgmap v967: 129 pgs: 129 active+clean; 426 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 13 KiB/s rd, 8.7 MiB/s wr, 426 op/s 2024-05-08T20:31:40.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:40 smithi083 ceph-mon[38695]: pgmap v967: 129 pgs: 129 active+clean; 426 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 13 KiB/s rd, 8.7 MiB/s wr, 426 op/s 2024-05-08T20:31:40.573 INFO:tasks.workunit.client.1.smithi188.stdout: 10 260699 22.49 MB/sec execute 484 sec latency 131.366 ms 2024-05-08T20:31:40.785 INFO:tasks.workunit.client.0.smithi083.stdout: 10 498719 44.50 MB/sec execute 484 sec latency 619.761 ms 2024-05-08T20:31:41.574 INFO:tasks.workunit.client.1.smithi188.stdout: 10 261029 22.47 MB/sec execute 485 sec latency 90.399 ms 2024-05-08T20:31:41.785 INFO:tasks.workunit.client.0.smithi083.stdout: 10 499089 44.44 MB/sec execute 485 sec latency 742.545 ms 2024-05-08T20:31:42.351 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:42 smithi188 ceph-mon[48761]: pgmap v968: 129 pgs: 129 active+clean; 427 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 534 op/s 2024-05-08T20:31:42.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:42 smithi083 ceph-mon[38695]: pgmap v968: 129 pgs: 129 active+clean; 427 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 534 op/s 2024-05-08T20:31:42.574 INFO:tasks.workunit.client.1.smithi188.stdout: 10 261334 22.46 MB/sec execute 486 sec latency 133.567 ms 2024-05-08T20:31:42.785 INFO:tasks.workunit.client.0.smithi083.stdout: 10 499267 44.36 MB/sec execute 486 sec latency 537.033 ms 2024-05-08T20:31:43.574 INFO:tasks.workunit.client.1.smithi188.stdout: 10 261615 22.43 MB/sec execute 487 sec latency 81.660 ms 2024-05-08T20:31:43.785 INFO:tasks.workunit.client.0.smithi083.stdout: 10 499529 44.28 MB/sec execute 487 sec latency 501.184 ms 2024-05-08T20:31:44.348 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:44 smithi188 ceph-mon[48761]: pgmap v969: 129 pgs: 129 active+clean; 433 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 432 op/s 2024-05-08T20:31:44.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:44 smithi083 ceph-mon[38695]: pgmap v969: 129 pgs: 129 active+clean; 433 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 432 op/s 2024-05-08T20:31:44.574 INFO:tasks.workunit.client.1.smithi188.stdout: 10 261972 22.43 MB/sec execute 488 sec latency 174.503 ms 2024-05-08T20:31:44.785 INFO:tasks.workunit.client.0.smithi083.stdout: 10 500692 44.36 MB/sec execute 488 sec latency 468.185 ms 2024-05-08T20:31:45.574 INFO:tasks.workunit.client.1.smithi188.stdout: 10 262392 22.43 MB/sec execute 489 sec latency 64.579 ms 2024-05-08T20:31:45.785 INFO:tasks.workunit.client.0.smithi083.stdout: 10 501682 44.37 MB/sec execute 489 sec latency 108.414 ms 2024-05-08T20:31:46.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:46 smithi188 ceph-mon[48761]: pgmap v970: 129 pgs: 129 active+clean; 422 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 9.6 MiB/s wr, 471 op/s 2024-05-08T20:31:46.448 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:46 smithi083 ceph-mon[38695]: pgmap v970: 129 pgs: 129 active+clean; 422 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 9.6 MiB/s wr, 471 op/s 2024-05-08T20:31:46.574 INFO:tasks.workunit.client.1.smithi188.stdout: 10 262876 22.42 MB/sec execute 490 sec latency 63.635 ms 2024-05-08T20:31:46.785 INFO:tasks.workunit.client.0.smithi083.stdout: 10 501906 44.29 MB/sec execute 490 sec latency 113.910 ms 2024-05-08T20:31:47.574 INFO:tasks.workunit.client.1.smithi188.stdout: 10 263248 22.43 MB/sec execute 491 sec latency 115.982 ms 2024-05-08T20:31:47.785 INFO:tasks.workunit.client.0.smithi083.stdout: 10 503355 44.33 MB/sec execute 491 sec latency 130.903 ms 2024-05-08T20:31:48.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:48 smithi188 ceph-mon[48761]: pgmap v971: 129 pgs: 129 active+clean; 419 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 18 KiB/s rd, 9.7 MiB/s wr, 480 op/s 2024-05-08T20:31:48.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:48 smithi083 ceph-mon[38695]: pgmap v971: 129 pgs: 129 active+clean; 419 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 18 KiB/s rd, 9.7 MiB/s wr, 480 op/s 2024-05-08T20:31:48.574 INFO:tasks.workunit.client.1.smithi188.stdout: 10 263740 22.45 MB/sec execute 492 sec latency 75.151 ms 2024-05-08T20:31:48.785 INFO:tasks.workunit.client.0.smithi083.stdout: 10 504722 44.44 MB/sec execute 492 sec latency 125.290 ms 2024-05-08T20:31:49.574 INFO:tasks.workunit.client.1.smithi188.stdout: 10 264122 22.45 MB/sec execute 493 sec latency 105.129 ms 2024-05-08T20:31:49.785 INFO:tasks.workunit.client.0.smithi083.stdout: 10 505270 44.40 MB/sec execute 493 sec latency 122.042 ms 2024-05-08T20:31:50.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:50 smithi188 ceph-mon[48761]: pgmap v972: 129 pgs: 129 active+clean; 421 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 407 op/s 2024-05-08T20:31:50.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:50 smithi083 ceph-mon[38695]: pgmap v972: 129 pgs: 129 active+clean; 421 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 407 op/s 2024-05-08T20:31:50.574 INFO:tasks.workunit.client.1.smithi188.stdout: 10 264497 22.44 MB/sec execute 494 sec latency 91.504 ms 2024-05-08T20:31:50.785 INFO:tasks.workunit.client.0.smithi083.stdout: 10 505761 44.36 MB/sec execute 494 sec latency 123.609 ms 2024-05-08T20:31:51.574 INFO:tasks.workunit.client.1.smithi188.stdout: 10 264891 22.44 MB/sec execute 495 sec latency 117.570 ms 2024-05-08T20:31:51.785 INFO:tasks.workunit.client.0.smithi083.stdout: 10 507137 44.41 MB/sec execute 495 sec latency 159.966 ms 2024-05-08T20:31:52.336 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:52 smithi083 ceph-mon[38695]: pgmap v973: 129 pgs: 129 active+clean; 431 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 17 KiB/s rd, 12 MiB/s wr, 569 op/s 2024-05-08T20:31:52.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:52 smithi188 ceph-mon[48761]: pgmap v973: 129 pgs: 129 active+clean; 431 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 17 KiB/s rd, 12 MiB/s wr, 569 op/s 2024-05-08T20:31:52.574 INFO:tasks.workunit.client.1.smithi188.stdout: 10 265244 22.42 MB/sec execute 496 sec latency 108.458 ms 2024-05-08T20:31:52.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 508712 44.51 MB/sec execute 496 sec latency 129.080 ms 2024-05-08T20:31:53.574 INFO:tasks.workunit.client.1.smithi188.stdout: 10 265595 22.41 MB/sec execute 497 sec latency 115.045 ms 2024-05-08T20:31:53.785 INFO:tasks.workunit.client.0.smithi083.stdout: 10 508834 44.43 MB/sec execute 497 sec latency 133.634 ms 2024-05-08T20:31:54.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:54 smithi188 ceph-mon[48761]: pgmap v974: 129 pgs: 129 active+clean; 433 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.6 MiB/s wr, 395 op/s 2024-05-08T20:31:54.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:31:54.450 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:54 smithi083 ceph-mon[38695]: pgmap v974: 129 pgs: 129 active+clean; 433 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.6 MiB/s wr, 395 op/s 2024-05-08T20:31:54.451 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:31:54.575 INFO:tasks.workunit.client.1.smithi188.stdout: 10 265905 22.40 MB/sec execute 498 sec latency 122.512 ms 2024-05-08T20:31:54.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 509138 44.37 MB/sec execute 498 sec latency 183.887 ms 2024-05-08T20:31:55.575 INFO:tasks.workunit.client.1.smithi188.stdout: 10 266420 22.40 MB/sec execute 499 sec latency 38.342 ms 2024-05-08T20:31:55.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 510546 44.40 MB/sec execute 499 sec latency 133.287 ms 2024-05-08T20:31:56.084 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:56 smithi083 ceph-mon[38695]: pgmap v975: 129 pgs: 129 active+clean; 429 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 412 op/s 2024-05-08T20:31:56.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:56 smithi188 ceph-mon[48761]: pgmap v975: 129 pgs: 129 active+clean; 429 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 412 op/s 2024-05-08T20:31:56.574 INFO:tasks.workunit.client.1.smithi188.stdout: 10 266837 22.41 MB/sec execute 500 sec latency 109.991 ms 2024-05-08T20:31:56.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 511902 44.50 MB/sec execute 500 sec latency 134.700 ms 2024-05-08T20:31:57.574 INFO:tasks.workunit.client.1.smithi188.stdout: 10 267339 22.43 MB/sec execute 501 sec latency 109.552 ms 2024-05-08T20:31:57.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 512377 44.46 MB/sec execute 501 sec latency 143.885 ms 2024-05-08T20:31:58.084 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:31:58 smithi188 ceph-mon[48761]: pgmap v976: 129 pgs: 129 active+clean; 417 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 19 KiB/s rd, 10 MiB/s wr, 471 op/s 2024-05-08T20:31:58.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:31:58 smithi083 ceph-mon[38695]: pgmap v976: 129 pgs: 129 active+clean; 417 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 19 KiB/s rd, 10 MiB/s wr, 471 op/s 2024-05-08T20:31:58.575 INFO:tasks.workunit.client.1.smithi188.stdout: 10 267687 22.43 MB/sec execute 502 sec latency 129.372 ms 2024-05-08T20:31:58.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 512850 44.42 MB/sec execute 502 sec latency 131.310 ms 2024-05-08T20:31:59.575 INFO:tasks.workunit.client.1.smithi188.stdout: 10 268060 22.42 MB/sec execute 503 sec latency 88.355 ms 2024-05-08T20:31:59.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 513905 44.41 MB/sec execute 503 sec latency 334.397 ms 2024-05-08T20:32:00.349 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:00 smithi188 ceph-mon[48761]: pgmap v977: 129 pgs: 129 active+clean; 419 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 9.0 MiB/s wr, 432 op/s 2024-05-08T20:32:00.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:00 smithi083 ceph-mon[38695]: pgmap v977: 129 pgs: 129 active+clean; 419 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 9.0 MiB/s wr, 432 op/s 2024-05-08T20:32:00.575 INFO:tasks.workunit.client.1.smithi188.stdout: 10 268435 22.42 MB/sec execute 504 sec latency 117.600 ms 2024-05-08T20:32:00.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 514807 44.46 MB/sec execute 504 sec latency 301.691 ms 2024-05-08T20:32:01.575 INFO:tasks.workunit.client.1.smithi188.stdout: 10 268802 22.40 MB/sec execute 505 sec latency 79.446 ms 2024-05-08T20:32:01.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 515858 44.48 MB/sec execute 505 sec latency 137.768 ms 2024-05-08T20:32:02.349 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:02 smithi188 ceph-mon[48761]: pgmap v978: 129 pgs: 129 active+clean; 428 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 562 op/s 2024-05-08T20:32:02.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:02 smithi083 ceph-mon[38695]: pgmap v978: 129 pgs: 129 active+clean; 428 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 562 op/s 2024-05-08T20:32:02.575 INFO:tasks.workunit.client.1.smithi188.stdout: 10 269141 22.38 MB/sec execute 506 sec latency 136.123 ms 2024-05-08T20:32:02.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 516009 44.40 MB/sec execute 506 sec latency 136.891 ms 2024-05-08T20:32:03.576 INFO:tasks.workunit.client.1.smithi188.stdout: 10 269578 22.40 MB/sec execute 507 sec latency 88.418 ms 2024-05-08T20:32:03.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 517107 44.41 MB/sec execute 507 sec latency 141.772 ms 2024-05-08T20:32:04.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:04 smithi083 ceph-mon[38695]: pgmap v979: 129 pgs: 129 active+clean; 425 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 13 KiB/s rd, 8.6 MiB/s wr, 398 op/s 2024-05-08T20:32:04.577 INFO:tasks.workunit.client.1.smithi188.stdout: 10 269978 22.38 MB/sec execute 508 sec latency 79.768 ms 2024-05-08T20:32:04.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:04 smithi188 ceph-mon[48761]: pgmap v979: 129 pgs: 129 active+clean; 425 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 13 KiB/s rd, 8.6 MiB/s wr, 398 op/s 2024-05-08T20:32:04.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 518426 44.49 MB/sec execute 508 sec latency 144.200 ms 2024-05-08T20:32:05.577 INFO:tasks.workunit.client.1.smithi188.stdout: 10 270336 22.39 MB/sec execute 509 sec latency 158.832 ms 2024-05-08T20:32:05.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 519368 44.50 MB/sec execute 509 sec latency 302.287 ms 2024-05-08T20:32:06.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:06 smithi083 ceph-mon[38695]: pgmap v980: 129 pgs: 129 active+clean; 421 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 13 KiB/s rd, 9.8 MiB/s wr, 432 op/s 2024-05-08T20:32:06.577 INFO:tasks.workunit.client.1.smithi188.stdout: 10 270889 22.41 MB/sec execute 510 sec latency 91.088 ms 2024-05-08T20:32:06.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:06 smithi188 ceph-mon[48761]: pgmap v980: 129 pgs: 129 active+clean; 421 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 13 KiB/s rd, 9.8 MiB/s wr, 432 op/s 2024-05-08T20:32:06.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 519476 44.42 MB/sec execute 510 sec latency 374.911 ms 2024-05-08T20:32:07.578 INFO:tasks.workunit.client.1.smithi188.stdout: 10 271281 22.41 MB/sec execute 511 sec latency 106.244 ms 2024-05-08T20:32:07.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 520251 44.43 MB/sec execute 511 sec latency 129.894 ms 2024-05-08T20:32:08.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:08 smithi188 ceph-mon[48761]: pgmap v981: 129 pgs: 129 active+clean; 414 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 19 KiB/s rd, 10 MiB/s wr, 530 op/s 2024-05-08T20:32:08.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:08 smithi083 ceph-mon[38695]: pgmap v981: 129 pgs: 129 active+clean; 414 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 19 KiB/s rd, 10 MiB/s wr, 530 op/s 2024-05-08T20:32:08.577 INFO:tasks.workunit.client.1.smithi188.stdout: 10 271662 22.41 MB/sec execute 512 sec latency 92.443 ms 2024-05-08T20:32:08.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 521631 44.48 MB/sec execute 512 sec latency 127.302 ms 2024-05-08T20:32:09.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:32:09.577 INFO:tasks.workunit.client.1.smithi188.stdout: 10 272013 22.40 MB/sec execute 513 sec latency 85.272 ms 2024-05-08T20:32:09.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:32:09.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 522909 44.53 MB/sec execute 513 sec latency 120.372 ms 2024-05-08T20:32:10.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:10 smithi083 ceph-mon[38695]: pgmap v982: 129 pgs: 129 active+clean; 414 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 13 KiB/s rd, 9.0 MiB/s wr, 447 op/s 2024-05-08T20:32:10.578 INFO:tasks.workunit.client.1.smithi188.stdout: 10 272328 22.38 MB/sec execute 514 sec latency 147.838 ms 2024-05-08T20:32:10.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:10 smithi188 ceph-mon[48761]: pgmap v982: 129 pgs: 129 active+clean; 414 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 13 KiB/s rd, 9.0 MiB/s wr, 447 op/s 2024-05-08T20:32:10.787 INFO:tasks.workunit.client.0.smithi083.stdout: 10 523038 44.45 MB/sec execute 514 sec latency 112.158 ms 2024-05-08T20:32:11.578 INFO:tasks.workunit.client.1.smithi188.stdout: 10 272624 22.36 MB/sec execute 515 sec latency 121.954 ms 2024-05-08T20:32:11.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 523903 44.45 MB/sec execute 515 sec latency 259.295 ms 2024-05-08T20:32:12.448 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:12 smithi083 ceph-mon[38695]: pgmap v983: 129 pgs: 129 active+clean; 420 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 543 op/s 2024-05-08T20:32:12.578 INFO:tasks.workunit.client.1.smithi188.stdout: 10 273017 22.37 MB/sec execute 516 sec latency 105.236 ms 2024-05-08T20:32:12.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:12 smithi188 ceph-mon[48761]: pgmap v983: 129 pgs: 129 active+clean; 420 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 543 op/s 2024-05-08T20:32:12.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 525236 44.51 MB/sec execute 516 sec latency 142.606 ms 2024-05-08T20:32:13.578 INFO:tasks.workunit.client.1.smithi188.stdout: 10 273450 22.36 MB/sec execute 517 sec latency 129.764 ms 2024-05-08T20:32:13.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 526460 44.55 MB/sec execute 517 sec latency 147.127 ms 2024-05-08T20:32:14.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:14 smithi083 ceph-mon[38695]: pgmap v984: 129 pgs: 129 active+clean; 418 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 385 op/s 2024-05-08T20:32:14.578 INFO:tasks.workunit.client.1.smithi188.stdout: 10 273850 22.36 MB/sec execute 518 sec latency 110.714 ms 2024-05-08T20:32:14.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:14 smithi188 ceph-mon[48761]: pgmap v984: 129 pgs: 129 active+clean; 418 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 385 op/s 2024-05-08T20:32:14.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 526589 44.47 MB/sec execute 518 sec latency 114.154 ms 2024-05-08T20:32:15.578 INFO:tasks.workunit.client.1.smithi188.stdout: 10 274425 22.39 MB/sec execute 519 sec latency 34.288 ms 2024-05-08T20:32:15.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 527747 44.49 MB/sec execute 519 sec latency 116.283 ms 2024-05-08T20:32:16.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:16 smithi083 ceph-mon[38695]: pgmap v985: 129 pgs: 129 active+clean; 424 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 498 op/s 2024-05-08T20:32:16.578 INFO:tasks.workunit.client.1.smithi188.stdout: 10 274790 22.39 MB/sec execute 520 sec latency 87.824 ms 2024-05-08T20:32:16.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:16 smithi188 ceph-mon[48761]: pgmap v985: 129 pgs: 129 active+clean; 424 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 498 op/s 2024-05-08T20:32:16.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 528643 44.52 MB/sec execute 520 sec latency 401.148 ms 2024-05-08T20:32:17.578 INFO:tasks.workunit.client.1.smithi188.stdout: 10 275118 22.38 MB/sec execute 521 sec latency 113.634 ms 2024-05-08T20:32:17.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 528832 44.46 MB/sec execute 521 sec latency 538.576 ms 2024-05-08T20:32:18.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:18 smithi188 ceph-mon[48761]: pgmap v986: 129 pgs: 129 active+clean; 412 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 18 KiB/s rd, 10 MiB/s wr, 513 op/s 2024-05-08T20:32:18.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:18 smithi083 ceph-mon[38695]: pgmap v986: 129 pgs: 129 active+clean; 412 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 18 KiB/s rd, 10 MiB/s wr, 513 op/s 2024-05-08T20:32:18.578 INFO:tasks.workunit.client.1.smithi188.stdout: 10 275436 22.37 MB/sec execute 522 sec latency 91.293 ms 2024-05-08T20:32:18.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 529055 44.40 MB/sec execute 522 sec latency 394.342 ms 2024-05-08T20:32:19.580 INFO:tasks.workunit.client.1.smithi188.stdout: 10 275736 22.35 MB/sec execute 523 sec latency 103.433 ms 2024-05-08T20:32:19.786 INFO:tasks.workunit.client.0.smithi083.stdout: 10 529360 44.34 MB/sec execute 523 sec latency 426.316 ms 2024-05-08T20:32:20.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:20 smithi083 ceph-mon[38695]: pgmap v987: 129 pgs: 129 active+clean; 414 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 387 op/s 2024-05-08T20:32:20.582 INFO:tasks.workunit.client.1.smithi188.stdout: 10 276065 22.33 MB/sec execute 524 sec latency 97.594 ms 2024-05-08T20:32:20.598 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:20 smithi188 ceph-mon[48761]: pgmap v987: 129 pgs: 129 active+clean; 414 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 387 op/s 2024-05-08T20:32:20.787 INFO:tasks.workunit.client.0.smithi083.stdout: 10 529990 44.32 MB/sec execute 524 sec latency 555.673 ms 2024-05-08T20:32:21.583 INFO:tasks.workunit.client.1.smithi188.stdout: 10 276390 22.33 MB/sec execute 525 sec latency 111.166 ms 2024-05-08T20:32:21.787 INFO:tasks.workunit.client.0.smithi083.stdout: 10 530154 44.25 MB/sec execute 525 sec latency 116.548 ms 2024-05-08T20:32:22.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:22 smithi083 ceph-mon[38695]: pgmap v988: 129 pgs: 129 active+clean; 414 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 543 op/s 2024-05-08T20:32:22.585 INFO:tasks.workunit.client.1.smithi188.stdout: 10 276737 22.32 MB/sec execute 526 sec latency 105.318 ms 2024-05-08T20:32:22.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:22 smithi188 ceph-mon[48761]: pgmap v988: 129 pgs: 129 active+clean; 414 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 543 op/s 2024-05-08T20:32:22.787 INFO:tasks.workunit.client.0.smithi083.stdout: 10 531174 44.25 MB/sec execute 526 sec latency 143.265 ms 2024-05-08T20:32:23.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 277114 22.31 MB/sec execute 527 sec latency 99.609 ms 2024-05-08T20:32:23.787 INFO:tasks.workunit.client.0.smithi083.stdout: 10 532486 44.32 MB/sec execute 527 sec latency 125.571 ms 2024-05-08T20:32:24.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:24 smithi083 ceph-mon[38695]: pgmap v989: 129 pgs: 129 active+clean; 413 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.1 MiB/s wr, 411 op/s 2024-05-08T20:32:24.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:32:24.585 INFO:tasks.workunit.client.1.smithi188.stdout: 10 277393 22.30 MB/sec execute 528 sec latency 124.666 ms 2024-05-08T20:32:24.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:24 smithi188 ceph-mon[48761]: pgmap v989: 129 pgs: 129 active+clean; 413 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.1 MiB/s wr, 411 op/s 2024-05-08T20:32:24.598 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:24 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:32:24.787 INFO:tasks.workunit.client.0.smithi083.stdout: 10 533603 44.35 MB/sec execute 528 sec latency 123.241 ms 2024-05-08T20:32:25.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 277851 22.32 MB/sec execute 529 sec latency 67.175 ms 2024-05-08T20:32:25.787 INFO:tasks.workunit.client.0.smithi083.stdout: 10 533784 44.28 MB/sec execute 529 sec latency 125.602 ms 2024-05-08T20:32:26.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:26 smithi083 ceph-mon[38695]: pgmap v990: 129 pgs: 129 active+clean; 410 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 483 op/s 2024-05-08T20:32:26.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 278267 22.32 MB/sec execute 530 sec latency 95.053 ms 2024-05-08T20:32:26.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:26 smithi188 ceph-mon[48761]: pgmap v990: 129 pgs: 129 active+clean; 410 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 483 op/s 2024-05-08T20:32:26.787 INFO:tasks.workunit.client.0.smithi083.stdout: 10 534869 44.29 MB/sec execute 530 sec latency 208.486 ms 2024-05-08T20:32:27.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 278681 22.32 MB/sec execute 531 sec latency 88.320 ms 2024-05-08T20:32:27.787 INFO:tasks.workunit.client.0.smithi083.stdout: 10 535957 44.34 MB/sec execute 531 sec latency 139.635 ms 2024-05-08T20:32:28.148 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:28 smithi083 ceph-mon[38695]: pgmap v991: 129 pgs: 129 active+clean; 412 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 17 KiB/s rd, 9.0 MiB/s wr, 397 op/s 2024-05-08T20:32:28.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 279034 22.31 MB/sec execute 532 sec latency 124.336 ms 2024-05-08T20:32:28.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:28 smithi188 ceph-mon[48761]: pgmap v991: 129 pgs: 129 active+clean; 412 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 17 KiB/s rd, 9.0 MiB/s wr, 397 op/s 2024-05-08T20:32:28.787 INFO:tasks.workunit.client.0.smithi083.stdout: 10 537131 44.38 MB/sec execute 532 sec latency 126.677 ms 2024-05-08T20:32:29.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 279392 22.30 MB/sec execute 533 sec latency 84.072 ms 2024-05-08T20:32:29.787 INFO:tasks.workunit.client.0.smithi083.stdout: 10 537263 44.30 MB/sec execute 533 sec latency 129.307 ms 2024-05-08T20:32:30.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:30 smithi083 ceph-mon[38695]: pgmap v992: 129 pgs: 129 active+clean; 418 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 402 op/s 2024-05-08T20:32:30.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 279740 22.29 MB/sec execute 534 sec latency 156.355 ms 2024-05-08T20:32:30.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:30 smithi188 ceph-mon[48761]: pgmap v992: 129 pgs: 129 active+clean; 418 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 402 op/s 2024-05-08T20:32:30.790 INFO:tasks.workunit.client.0.smithi083.stdout: 10 538413 44.31 MB/sec execute 534 sec latency 122.304 ms 2024-05-08T20:32:31.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 280209 22.30 MB/sec execute 535 sec latency 90.714 ms 2024-05-08T20:32:31.789 INFO:tasks.workunit.client.0.smithi083.stdout: 10 539001 44.29 MB/sec execute 535 sec latency 339.648 ms 2024-05-08T20:32:32.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:32 smithi083 ceph-mon[38695]: pgmap v993: 129 pgs: 129 active+clean; 421 MiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 571 op/s 2024-05-08T20:32:32.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 280683 22.29 MB/sec execute 536 sec latency 116.868 ms 2024-05-08T20:32:32.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:32 smithi188 ceph-mon[48761]: pgmap v993: 129 pgs: 129 active+clean; 421 MiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 571 op/s 2024-05-08T20:32:32.789 INFO:tasks.workunit.client.0.smithi083.stdout: 10 540301 44.37 MB/sec execute 536 sec latency 308.612 ms 2024-05-08T20:32:33.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 281039 22.29 MB/sec execute 537 sec latency 93.813 ms 2024-05-08T20:32:33.789 INFO:tasks.workunit.client.0.smithi083.stdout: 10 540752 44.32 MB/sec execute 537 sec latency 126.439 ms 2024-05-08T20:32:34.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:34 smithi083 ceph-mon[38695]: pgmap v994: 129 pgs: 129 active+clean; 419 MiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 13 KiB/s rd, 8.1 MiB/s wr, 402 op/s 2024-05-08T20:32:34.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:34 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:32:34.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:34 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:32:34.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:34 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:32:34.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 281462 22.30 MB/sec execute 538 sec latency 78.591 ms 2024-05-08T20:32:34.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:34 smithi188 ceph-mon[48761]: pgmap v994: 129 pgs: 129 active+clean; 419 MiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 13 KiB/s rd, 8.1 MiB/s wr, 402 op/s 2024-05-08T20:32:34.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:34 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:32:34.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:34 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:32:34.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:34 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:32:34.789 INFO:tasks.workunit.client.0.smithi083.stdout: 10 541149 44.27 MB/sec execute 538 sec latency 117.203 ms 2024-05-08T20:32:35.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:32:35.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:32:35.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:32:35.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 281855 22.31 MB/sec execute 539 sec latency 117.039 ms 2024-05-08T20:32:35.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:32:35.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:32:35.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:32:35.789 INFO:tasks.workunit.client.0.smithi083.stdout: 10 542550 44.31 MB/sec execute 539 sec latency 142.337 ms 2024-05-08T20:32:36.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:36 smithi083 ceph-mon[38695]: pgmap v995: 129 pgs: 129 active+clean; 417 MiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 13 KiB/s rd, 9.3 MiB/s wr, 422 op/s 2024-05-08T20:32:36.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 282297 22.31 MB/sec execute 540 sec latency 128.973 ms 2024-05-08T20:32:36.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:36 smithi188 ceph-mon[48761]: pgmap v995: 129 pgs: 129 active+clean; 417 MiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 13 KiB/s rd, 9.3 MiB/s wr, 422 op/s 2024-05-08T20:32:36.789 INFO:tasks.workunit.client.0.smithi083.stdout: 10 543755 44.38 MB/sec execute 540 sec latency 154.714 ms 2024-05-08T20:32:37.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 282648 22.30 MB/sec execute 541 sec latency 117.588 ms 2024-05-08T20:32:37.789 INFO:tasks.workunit.client.0.smithi083.stdout: 10 544236 44.35 MB/sec execute 541 sec latency 377.252 ms 2024-05-08T20:32:38.418 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:38 smithi188 ceph-mon[48761]: pgmap v996: 129 pgs: 129 active+clean; 418 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 470 op/s 2024-05-08T20:32:38.446 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:38 smithi083 ceph-mon[38695]: pgmap v996: 129 pgs: 129 active+clean; 418 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 470 op/s 2024-05-08T20:32:38.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 283017 22.28 MB/sec execute 542 sec latency 125.836 ms 2024-05-08T20:32:38.789 INFO:tasks.workunit.client.0.smithi083.stdout: 10 544405 44.27 MB/sec execute 542 sec latency 108.851 ms 2024-05-08T20:32:39.448 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:32:39.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 283406 22.29 MB/sec execute 543 sec latency 115.023 ms 2024-05-08T20:32:39.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:32:39.789 INFO:tasks.workunit.client.0.smithi083.stdout: 10 545646 44.29 MB/sec execute 543 sec latency 148.856 ms 2024-05-08T20:32:40.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:40 smithi083 ceph-mon[38695]: pgmap v997: 129 pgs: 129 active+clean; 424 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 9.6 MiB/s wr, 462 op/s 2024-05-08T20:32:40.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 283843 22.29 MB/sec execute 544 sec latency 97.815 ms 2024-05-08T20:32:40.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:40 smithi188 ceph-mon[48761]: pgmap v997: 129 pgs: 129 active+clean; 424 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 9.6 MiB/s wr, 462 op/s 2024-05-08T20:32:40.789 INFO:tasks.workunit.client.0.smithi083.stdout: 10 546830 44.36 MB/sec execute 544 sec latency 126.969 ms 2024-05-08T20:32:41.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 284272 22.28 MB/sec execute 545 sec latency 91.178 ms 2024-05-08T20:32:41.789 INFO:tasks.workunit.client.0.smithi083.stdout: 10 547806 44.37 MB/sec execute 545 sec latency 117.288 ms 2024-05-08T20:32:42.449 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:42 smithi083 ceph-mon[38695]: pgmap v998: 129 pgs: 129 active+clean; 415 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 565 op/s 2024-05-08T20:32:42.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 284701 22.30 MB/sec execute 546 sec latency 98.491 ms 2024-05-08T20:32:42.599 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:42 smithi188 ceph-mon[48761]: pgmap v998: 129 pgs: 129 active+clean; 415 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 565 op/s 2024-05-08T20:32:42.789 INFO:tasks.workunit.client.0.smithi083.stdout: 10 547917 44.30 MB/sec execute 546 sec latency 314.645 ms 2024-05-08T20:32:43.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 285132 22.29 MB/sec execute 547 sec latency 91.326 ms 2024-05-08T20:32:43.789 INFO:tasks.workunit.client.0.smithi083.stdout: 10 549052 44.31 MB/sec execute 547 sec latency 129.242 ms 2024-05-08T20:32:44.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:44 smithi083 ceph-mon[38695]: pgmap v999: 129 pgs: 129 active+clean; 411 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 411 op/s 2024-05-08T20:32:44.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 285455 22.29 MB/sec execute 548 sec latency 121.694 ms 2024-05-08T20:32:44.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:44 smithi188 ceph-mon[48761]: pgmap v999: 129 pgs: 129 active+clean; 411 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 411 op/s 2024-05-08T20:32:44.789 INFO:tasks.workunit.client.0.smithi083.stdout: 10 550216 44.37 MB/sec execute 548 sec latency 133.175 ms 2024-05-08T20:32:45.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 285855 22.29 MB/sec execute 549 sec latency 108.477 ms 2024-05-08T20:32:45.789 INFO:tasks.workunit.client.0.smithi083.stdout: 10 551343 44.40 MB/sec execute 549 sec latency 132.227 ms 2024-05-08T20:32:46.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:46 smithi083 ceph-mon[38695]: pgmap v1000: 129 pgs: 129 active+clean; 410 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 11 KiB/s rd, 9.9 MiB/s wr, 445 op/s 2024-05-08T20:32:46.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 286189 22.28 MB/sec execute 550 sec latency 120.263 ms 2024-05-08T20:32:46.601 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:46 smithi188 ceph-mon[48761]: pgmap v1000: 129 pgs: 129 active+clean; 410 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 11 KiB/s rd, 9.9 MiB/s wr, 445 op/s 2024-05-08T20:32:46.789 INFO:tasks.workunit.client.0.smithi083.stdout: 10 551498 44.32 MB/sec execute 550 sec latency 124.051 ms 2024-05-08T20:32:47.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 286545 22.27 MB/sec execute 551 sec latency 148.891 ms 2024-05-08T20:32:47.789 INFO:tasks.workunit.client.0.smithi083.stdout: 10 552740 44.34 MB/sec execute 551 sec latency 125.661 ms 2024-05-08T20:32:48.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:48 smithi083 ceph-mon[38695]: pgmap v1001: 129 pgs: 129 active+clean; 422 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 558 op/s 2024-05-08T20:32:48.583 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:48 smithi188 ceph-mon[48761]: pgmap v1001: 129 pgs: 129 active+clean; 422 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 558 op/s 2024-05-08T20:32:48.590 INFO:tasks.workunit.client.1.smithi188.stdout: 10 286888 22.25 MB/sec execute 552 sec latency 113.156 ms 2024-05-08T20:32:48.789 INFO:tasks.workunit.client.0.smithi083.stdout: 10 553875 44.40 MB/sec execute 552 sec latency 142.863 ms 2024-05-08T20:32:49.587 INFO:tasks.workunit.client.1.smithi188.stdout: 10 287331 22.27 MB/sec execute 553 sec latency 107.171 ms 2024-05-08T20:32:49.789 INFO:tasks.workunit.client.0.smithi083.stdout: 10 554895 44.42 MB/sec execute 553 sec latency 127.131 ms 2024-05-08T20:32:50.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:50 smithi083 ceph-mon[38695]: pgmap v1002: 129 pgs: 129 active+clean; 424 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 11 KiB/s rd, 9.3 MiB/s wr, 447 op/s 2024-05-08T20:32:50.586 INFO:tasks.workunit.client.1.smithi188.stdout: 10 287797 22.26 MB/sec execute 554 sec latency 108.991 ms 2024-05-08T20:32:50.598 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:50 smithi188 ceph-mon[48761]: pgmap v1002: 129 pgs: 129 active+clean; 424 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 11 KiB/s rd, 9.3 MiB/s wr, 447 op/s 2024-05-08T20:32:50.789 INFO:tasks.workunit.client.0.smithi083.stdout: 10 555023 44.35 MB/sec execute 554 sec latency 141.784 ms 2024-05-08T20:32:51.587 INFO:tasks.workunit.client.1.smithi188.stdout: 10 288124 22.26 MB/sec execute 555 sec latency 111.537 ms 2024-05-08T20:32:51.789 INFO:tasks.workunit.client.0.smithi083.stdout: 10 556195 44.36 MB/sec execute 555 sec latency 128.204 ms 2024-05-08T20:32:52.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:52 smithi083 ceph-mon[38695]: pgmap v1003: 129 pgs: 129 active+clean; 413 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 572 op/s 2024-05-08T20:32:52.588 INFO:tasks.workunit.client.1.smithi188.stdout: 10 288581 22.27 MB/sec execute 556 sec latency 132.371 ms 2024-05-08T20:32:52.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:52 smithi188 ceph-mon[48761]: pgmap v1003: 129 pgs: 129 active+clean; 413 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 572 op/s 2024-05-08T20:32:52.790 INFO:tasks.workunit.client.0.smithi083.stdout: 10 557422 44.43 MB/sec execute 556 sec latency 143.133 ms 2024-05-08T20:32:53.588 INFO:tasks.workunit.client.1.smithi188.stdout: 10 288944 22.27 MB/sec execute 557 sec latency 62.520 ms 2024-05-08T20:32:53.790 INFO:tasks.workunit.client.0.smithi083.stdout: 10 558029 44.41 MB/sec execute 557 sec latency 641.447 ms 2024-05-08T20:32:54.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:54 smithi083 ceph-mon[38695]: pgmap v1004: 129 pgs: 129 active+clean; 412 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 11 KiB/s rd, 8.2 MiB/s wr, 417 op/s 2024-05-08T20:32:54.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:32:54.588 INFO:tasks.workunit.client.1.smithi188.stdout: 10 289365 22.27 MB/sec execute 558 sec latency 62.967 ms 2024-05-08T20:32:54.599 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:54 smithi188 ceph-mon[48761]: pgmap v1004: 129 pgs: 129 active+clean; 412 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 11 KiB/s rd, 8.2 MiB/s wr, 417 op/s 2024-05-08T20:32:54.599 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:32:54.790 INFO:tasks.workunit.client.0.smithi083.stdout: 10 558385 44.37 MB/sec execute 558 sec latency 697.179 ms 2024-05-08T20:32:55.588 INFO:tasks.workunit.client.1.smithi188.stdout: 10 289724 22.27 MB/sec execute 559 sec latency 156.310 ms 2024-05-08T20:32:55.790 INFO:tasks.workunit.client.0.smithi083.stdout: 10 558390 44.29 MB/sec execute 559 sec latency 620.390 ms 2024-05-08T20:32:56.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:56 smithi083 ceph-mon[38695]: pgmap v1005: 129 pgs: 129 active+clean; 419 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 429 op/s 2024-05-08T20:32:56.588 INFO:tasks.workunit.client.1.smithi188.stdout: 10 290033 22.25 MB/sec execute 560 sec latency 95.116 ms 2024-05-08T20:32:56.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:56 smithi188 ceph-mon[48761]: pgmap v1005: 129 pgs: 129 active+clean; 419 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 429 op/s 2024-05-08T20:32:56.790 INFO:tasks.workunit.client.0.smithi083.stdout: 10 558442 44.21 MB/sec execute 560 sec latency 605.698 ms 2024-05-08T20:32:57.588 INFO:tasks.workunit.client.1.smithi188.stdout: 10 290396 22.23 MB/sec execute 561 sec latency 115.516 ms 2024-05-08T20:32:57.790 INFO:tasks.workunit.client.0.smithi083.stdout: 10 558663 44.14 MB/sec execute 561 sec latency 124.226 ms 2024-05-08T20:32:58.380 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:32:58 smithi083 ceph-mon[38695]: pgmap v1006: 129 pgs: 129 active+clean; 425 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 491 op/s 2024-05-08T20:32:58.588 INFO:tasks.workunit.client.1.smithi188.stdout: 10 290800 22.24 MB/sec execute 562 sec latency 93.756 ms 2024-05-08T20:32:58.599 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:32:58 smithi188 ceph-mon[48761]: pgmap v1006: 129 pgs: 129 active+clean; 425 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 491 op/s 2024-05-08T20:32:58.790 INFO:tasks.workunit.client.0.smithi083.stdout: 10 560111 44.17 MB/sec execute 562 sec latency 148.763 ms 2024-05-08T20:32:59.589 INFO:tasks.workunit.client.1.smithi188.stdout: 10 291258 22.23 MB/sec execute 563 sec latency 123.018 ms 2024-05-08T20:32:59.790 INFO:tasks.workunit.client.0.smithi083.stdout: 10 561303 44.25 MB/sec execute 563 sec latency 137.425 ms 2024-05-08T20:33:00.589 INFO:tasks.workunit.client.1.smithi188.stdout: 10 291627 22.24 MB/sec execute 564 sec latency 100.297 ms 2024-05-08T20:33:00.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:00 smithi188 ceph-mon[48761]: pgmap v1007: 129 pgs: 129 active+clean; 422 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 361 op/s 2024-05-08T20:33:00.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:00 smithi083 ceph-mon[38695]: pgmap v1007: 129 pgs: 129 active+clean; 422 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 361 op/s 2024-05-08T20:33:00.790 INFO:tasks.workunit.client.0.smithi083.stdout: 10 562006 44.24 MB/sec execute 564 sec latency 118.921 ms 2024-05-08T20:33:01.589 INFO:tasks.workunit.client.1.smithi188.stdout: 10 292168 22.26 MB/sec execute 565 sec latency 84.147 ms 2024-05-08T20:33:01.790 INFO:tasks.workunit.client.0.smithi083.stdout: 10 562211 44.17 MB/sec execute 565 sec latency 108.931 ms 2024-05-08T20:33:02.589 INFO:tasks.workunit.client.1.smithi188.stdout: 10 292539 22.26 MB/sec execute 566 sec latency 110.057 ms 2024-05-08T20:33:02.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:02 smithi188 ceph-mon[48761]: pgmap v1008: 129 pgs: 129 active+clean; 412 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 528 op/s 2024-05-08T20:33:02.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:02 smithi083 ceph-mon[38695]: pgmap v1008: 129 pgs: 129 active+clean; 412 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 528 op/s 2024-05-08T20:33:02.790 INFO:tasks.workunit.client.0.smithi083.stdout: 10 563643 44.20 MB/sec execute 566 sec latency 146.789 ms 2024-05-08T20:33:03.589 INFO:tasks.workunit.client.1.smithi188.stdout: 10 292860 22.25 MB/sec execute 567 sec latency 121.479 ms 2024-05-08T20:33:03.790 INFO:tasks.workunit.client.0.smithi083.stdout: 10 564758 44.27 MB/sec execute 567 sec latency 133.791 ms 2024-05-08T20:33:04.589 INFO:tasks.workunit.client.1.smithi188.stdout: 10 293218 22.24 MB/sec execute 568 sec latency 112.321 ms 2024-05-08T20:33:04.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:04 smithi188 ceph-mon[48761]: pgmap v1009: 129 pgs: 129 active+clean; 412 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 374 op/s 2024-05-08T20:33:04.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:04 smithi083 ceph-mon[38695]: pgmap v1009: 129 pgs: 129 active+clean; 412 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 374 op/s 2024-05-08T20:33:04.790 INFO:tasks.workunit.client.0.smithi083.stdout: 10 565547 44.26 MB/sec execute 568 sec latency 113.482 ms 2024-05-08T20:33:05.589 INFO:tasks.workunit.client.1.smithi188.stdout: 10 293475 22.22 MB/sec execute 569 sec latency 116.980 ms 2024-05-08T20:33:05.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 565878 44.21 MB/sec execute 569 sec latency 114.653 ms 2024-05-08T20:33:06.589 INFO:tasks.workunit.client.1.smithi188.stdout: 10 293763 22.20 MB/sec execute 570 sec latency 116.802 ms 2024-05-08T20:33:06.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:06 smithi188 ceph-mon[48761]: pgmap v1010: 129 pgs: 129 active+clean; 418 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 431 op/s 2024-05-08T20:33:06.696 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:06 smithi083 ceph-mon[38695]: pgmap v1010: 129 pgs: 129 active+clean; 418 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 431 op/s 2024-05-08T20:33:06.790 INFO:tasks.workunit.client.0.smithi083.stdout: 10 567272 44.24 MB/sec execute 570 sec latency 149.055 ms 2024-05-08T20:33:07.589 INFO:tasks.workunit.client.1.smithi188.stdout: 10 294096 22.20 MB/sec execute 571 sec latency 72.649 ms 2024-05-08T20:33:07.790 INFO:tasks.workunit.client.0.smithi083.stdout: 10 567796 44.24 MB/sec execute 571 sec latency 401.380 ms 2024-05-08T20:33:08.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:08 smithi083 ceph-mon[38695]: pgmap v1011: 129 pgs: 129 active+clean; 424 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 522 op/s 2024-05-08T20:33:08.589 INFO:tasks.workunit.client.1.smithi188.stdout: 10 294476 22.20 MB/sec execute 572 sec latency 157.903 ms 2024-05-08T20:33:08.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:08 smithi188 ceph-mon[48761]: pgmap v1011: 129 pgs: 129 active+clean; 424 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 522 op/s 2024-05-08T20:33:08.790 INFO:tasks.workunit.client.0.smithi083.stdout: 10 568395 44.22 MB/sec execute 572 sec latency 376.896 ms 2024-05-08T20:33:09.227 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:09 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:33:09.589 INFO:tasks.workunit.client.1.smithi188.stdout: 10 294908 22.19 MB/sec execute 573 sec latency 107.185 ms 2024-05-08T20:33:09.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:09 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:33:09.790 INFO:tasks.workunit.client.0.smithi083.stdout: 10 569123 44.21 MB/sec execute 573 sec latency 113.121 ms 2024-05-08T20:33:10.589 INFO:tasks.workunit.client.1.smithi188.stdout: 10 295331 22.21 MB/sec execute 574 sec latency 88.187 ms 2024-05-08T20:33:10.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:10 smithi188 ceph-mon[48761]: pgmap v1012: 129 pgs: 129 active+clean; 423 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 13 KiB/s rd, 8.8 MiB/s wr, 480 op/s 2024-05-08T20:33:10.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:10 smithi083 ceph-mon[38695]: pgmap v1012: 129 pgs: 129 active+clean; 423 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 13 KiB/s rd, 8.8 MiB/s wr, 480 op/s 2024-05-08T20:33:10.790 INFO:tasks.workunit.client.0.smithi083.stdout: 10 569614 44.18 MB/sec execute 574 sec latency 122.444 ms 2024-05-08T20:33:11.589 INFO:tasks.workunit.client.1.smithi188.stdout: 10 295723 22.20 MB/sec execute 575 sec latency 95.766 ms 2024-05-08T20:33:11.790 INFO:tasks.workunit.client.0.smithi083.stdout: 10 570914 44.20 MB/sec execute 575 sec latency 139.682 ms 2024-05-08T20:33:12.589 INFO:tasks.workunit.client.1.smithi188.stdout: 10 296083 22.21 MB/sec execute 576 sec latency 70.523 ms 2024-05-08T20:33:12.598 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:12 smithi188 ceph-mon[48761]: pgmap v1013: 129 pgs: 129 active+clean; 413 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 569 op/s 2024-05-08T20:33:12.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:12 smithi083 ceph-mon[38695]: pgmap v1013: 129 pgs: 129 active+clean; 413 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 569 op/s 2024-05-08T20:33:12.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 572120 44.26 MB/sec execute 576 sec latency 134.596 ms 2024-05-08T20:33:13.589 INFO:tasks.workunit.client.1.smithi188.stdout: 10 296472 22.20 MB/sec execute 577 sec latency 92.357 ms 2024-05-08T20:33:13.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 572627 44.23 MB/sec execute 577 sec latency 249.859 ms 2024-05-08T20:33:14.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:14 smithi188 ceph-mon[48761]: pgmap v1014: 129 pgs: 129 active+clean; 419 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 431 op/s 2024-05-08T20:33:14.590 INFO:tasks.workunit.client.1.smithi188.stdout: 10 296801 22.20 MB/sec execute 578 sec latency 144.056 ms 2024-05-08T20:33:14.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:14 smithi083 ceph-mon[38695]: pgmap v1014: 129 pgs: 129 active+clean; 419 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 431 op/s 2024-05-08T20:33:14.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 572809 44.16 MB/sec execute 578 sec latency 333.254 ms 2024-05-08T20:33:15.590 INFO:tasks.workunit.client.1.smithi188.stdout: 10 297173 22.18 MB/sec execute 579 sec latency 80.875 ms 2024-05-08T20:33:15.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 574212 44.19 MB/sec execute 579 sec latency 137.979 ms 2024-05-08T20:33:16.590 INFO:tasks.workunit.client.1.smithi188.stdout: 10 297528 22.16 MB/sec execute 580 sec latency 105.874 ms 2024-05-08T20:33:16.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:16 smithi188 ceph-mon[48761]: pgmap v1015: 129 pgs: 129 active+clean; 423 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 459 op/s 2024-05-08T20:33:16.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:16 smithi083 ceph-mon[38695]: pgmap v1015: 129 pgs: 129 active+clean; 423 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 459 op/s 2024-05-08T20:33:16.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 575195 44.25 MB/sec execute 580 sec latency 117.428 ms 2024-05-08T20:33:17.590 INFO:tasks.workunit.client.1.smithi188.stdout: 10 297939 22.18 MB/sec execute 581 sec latency 114.781 ms 2024-05-08T20:33:17.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 576172 44.26 MB/sec execute 581 sec latency 133.180 ms 2024-05-08T20:33:18.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:18 smithi083 ceph-mon[38695]: pgmap v1016: 129 pgs: 129 active+clean; 424 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 17 KiB/s rd, 9.9 MiB/s wr, 485 op/s 2024-05-08T20:33:18.590 INFO:tasks.workunit.client.1.smithi188.stdout: 10 298373 22.17 MB/sec execute 582 sec latency 121.094 ms 2024-05-08T20:33:18.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:18 smithi188 ceph-mon[48761]: pgmap v1016: 129 pgs: 129 active+clean; 424 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 17 KiB/s rd, 9.9 MiB/s wr, 485 op/s 2024-05-08T20:33:18.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 576367 44.19 MB/sec execute 582 sec latency 125.151 ms 2024-05-08T20:33:19.590 INFO:tasks.workunit.client.1.smithi188.stdout: 10 298728 22.17 MB/sec execute 583 sec latency 69.273 ms 2024-05-08T20:33:19.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 577759 44.22 MB/sec execute 583 sec latency 141.570 ms 2024-05-08T20:33:20.590 INFO:tasks.workunit.client.1.smithi188.stdout: 10 299169 22.18 MB/sec execute 584 sec latency 90.602 ms 2024-05-08T20:33:20.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:20 smithi188 ceph-mon[48761]: pgmap v1017: 129 pgs: 129 active+clean; 423 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 380 op/s 2024-05-08T20:33:20.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:20 smithi083 ceph-mon[38695]: pgmap v1017: 129 pgs: 129 active+clean; 423 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 380 op/s 2024-05-08T20:33:20.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 578754 44.27 MB/sec execute 584 sec latency 139.575 ms 2024-05-08T20:33:21.590 INFO:tasks.workunit.client.1.smithi188.stdout: 10 299526 22.17 MB/sec execute 585 sec latency 115.379 ms 2024-05-08T20:33:21.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 579717 44.28 MB/sec execute 585 sec latency 128.522 ms 2024-05-08T20:33:22.592 INFO:tasks.workunit.client.1.smithi188.stdout: 10 299892 22.18 MB/sec execute 586 sec latency 123.325 ms 2024-05-08T20:33:22.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:22 smithi188 ceph-mon[48761]: pgmap v1018: 129 pgs: 129 active+clean; 424 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 501 op/s 2024-05-08T20:33:22.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:22 smithi083 ceph-mon[38695]: pgmap v1018: 129 pgs: 129 active+clean; 424 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 501 op/s 2024-05-08T20:33:22.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 579911 44.21 MB/sec execute 586 sec latency 134.128 ms 2024-05-08T20:33:23.592 INFO:tasks.workunit.client.1.smithi188.stdout: 10 300250 22.18 MB/sec execute 587 sec latency 175.387 ms 2024-05-08T20:33:23.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 581159 44.23 MB/sec execute 587 sec latency 142.176 ms 2024-05-08T20:33:24.592 INFO:tasks.workunit.client.1.smithi188.stdout: 10 300569 22.16 MB/sec execute 588 sec latency 144.757 ms 2024-05-08T20:33:24.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:24 smithi188 ceph-mon[48761]: pgmap v1019: 129 pgs: 129 active+clean; 420 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.6 MiB/s wr, 398 op/s 2024-05-08T20:33:24.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:24 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:33:24.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:24 smithi083 ceph-mon[38695]: pgmap v1019: 129 pgs: 129 active+clean; 420 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.6 MiB/s wr, 398 op/s 2024-05-08T20:33:24.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:24 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:33:24.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 582292 44.29 MB/sec execute 588 sec latency 137.972 ms 2024-05-08T20:33:25.594 INFO:tasks.workunit.client.1.smithi188.stdout: 10 300911 22.14 MB/sec execute 589 sec latency 115.198 ms 2024-05-08T20:33:25.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 583266 44.31 MB/sec execute 589 sec latency 122.648 ms 2024-05-08T20:33:26.594 INFO:tasks.workunit.client.1.smithi188.stdout: 10 301313 22.15 MB/sec execute 590 sec latency 121.212 ms 2024-05-08T20:33:26.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:26 smithi188 ceph-mon[48761]: pgmap v1020: 129 pgs: 129 active+clean; 424 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 431 op/s 2024-05-08T20:33:26.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:26 smithi083 ceph-mon[38695]: pgmap v1020: 129 pgs: 129 active+clean; 424 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 431 op/s 2024-05-08T20:33:26.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 583471 44.24 MB/sec execute 590 sec latency 128.594 ms 2024-05-08T20:33:27.594 INFO:tasks.workunit.client.1.smithi188.stdout: 10 301721 22.14 MB/sec execute 591 sec latency 107.958 ms 2024-05-08T20:33:27.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 584713 44.26 MB/sec execute 591 sec latency 140.978 ms 2024-05-08T20:33:28.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:28 smithi083 ceph-mon[38695]: pgmap v1021: 129 pgs: 129 active+clean; 425 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 489 op/s 2024-05-08T20:33:28.594 INFO:tasks.workunit.client.1.smithi188.stdout: 10 302099 22.14 MB/sec execute 592 sec latency 107.309 ms 2024-05-08T20:33:28.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:28 smithi188 ceph-mon[48761]: pgmap v1021: 129 pgs: 129 active+clean; 425 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 489 op/s 2024-05-08T20:33:28.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 585837 44.32 MB/sec execute 592 sec latency 142.726 ms 2024-05-08T20:33:29.595 INFO:tasks.workunit.client.1.smithi188.stdout: 10 302482 22.15 MB/sec execute 593 sec latency 73.315 ms 2024-05-08T20:33:29.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 586771 44.33 MB/sec execute 593 sec latency 318.498 ms 2024-05-08T20:33:30.595 INFO:tasks.workunit.client.1.smithi188.stdout: 10 302920 22.15 MB/sec execute 594 sec latency 79.640 ms 2024-05-08T20:33:30.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:30 smithi188 ceph-mon[48761]: pgmap v1022: 129 pgs: 129 active+clean; 418 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.6 MiB/s wr, 421 op/s 2024-05-08T20:33:30.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:30 smithi083 ceph-mon[38695]: pgmap v1022: 129 pgs: 129 active+clean; 418 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.6 MiB/s wr, 421 op/s 2024-05-08T20:33:30.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 586800 44.25 MB/sec execute 594 sec latency 603.149 ms 2024-05-08T20:33:31.595 INFO:tasks.workunit.client.1.smithi188.stdout: 10 303283 22.15 MB/sec execute 595 sec latency 142.241 ms 2024-05-08T20:33:31.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 586809 44.18 MB/sec execute 595 sec latency 578.198 ms 2024-05-08T20:33:32.595 INFO:tasks.workunit.client.1.smithi188.stdout: 10 303606 22.14 MB/sec execute 596 sec latency 153.400 ms 2024-05-08T20:33:32.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:32 smithi188 ceph-mon[48761]: pgmap v1023: 129 pgs: 129 active+clean; 419 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 515 op/s 2024-05-08T20:33:32.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:32 smithi083 ceph-mon[38695]: pgmap v1023: 129 pgs: 129 active+clean; 419 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 515 op/s 2024-05-08T20:33:32.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 586843 44.11 MB/sec execute 596 sec latency 554.898 ms 2024-05-08T20:33:33.594 INFO:tasks.workunit.client.1.smithi188.stdout: 10 303996 22.14 MB/sec execute 597 sec latency 120.118 ms 2024-05-08T20:33:33.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 587406 44.08 MB/sec execute 597 sec latency 135.077 ms 2024-05-08T20:33:34.283 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:34 smithi083 ceph-mon[38695]: pgmap v1024: 129 pgs: 129 active+clean; 422 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.1 MiB/s wr, 381 op/s 2024-05-08T20:33:34.596 INFO:tasks.workunit.client.1.smithi188.stdout: 10 304289 22.12 MB/sec execute 598 sec latency 118.113 ms 2024-05-08T20:33:34.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:34 smithi188 ceph-mon[48761]: pgmap v1024: 129 pgs: 129 active+clean; 422 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.1 MiB/s wr, 381 op/s 2024-05-08T20:33:34.792 INFO:tasks.workunit.client.0.smithi083.stdout: 10 588952 44.15 MB/sec execute 598 sec latency 109.376 ms 2024-05-08T20:33:35.596 INFO:tasks.workunit.client.1.smithi188.stdout: 10 304574 22.10 MB/sec execute 599 sec latency 152.405 ms 2024-05-08T20:33:35.602 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:33:35.602 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:33:35.602 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:33:35.603 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:35 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:33:35.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-08T20:33:35.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-08T20:33:35.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-08T20:33:35.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:35 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' 2024-05-08T20:33:35.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 590304 44.20 MB/sec execute 599 sec latency 125.472 ms 2024-05-08T20:33:36.596 INFO:tasks.workunit.client.1.smithi188.stdout: 10 cleanup 600 sec 2024-05-08T20:33:36.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:36 smithi188 ceph-mon[48761]: pgmap v1025: 129 pgs: 129 active+clean; 420 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 9.5 MiB/s wr, 408 op/s 2024-05-08T20:33:36.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:36 smithi083 ceph-mon[38695]: pgmap v1025: 129 pgs: 129 active+clean; 420 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 9.5 MiB/s wr, 408 op/s 2024-05-08T20:33:36.791 INFO:tasks.workunit.client.0.smithi083.stdout: 10 cleanup 600 sec 2024-05-08T20:33:36.820 INFO:tasks.workunit.client.0.smithi083.stdout: 0 cleanup 600 sec 2024-05-08T20:33:36.820 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-05-08T20:33:36.820 INFO:tasks.workunit.client.0.smithi083.stdout: Operation Count AvgLat MaxLat 2024-05-08T20:33:36.820 INFO:tasks.workunit.client.0.smithi083.stdout: ---------------------------------------- 2024-05-08T20:33:36.820 INFO:tasks.workunit.client.0.smithi083.stdout: NTCreateX 843956 0.018 80.454 2024-05-08T20:33:36.820 INFO:tasks.workunit.client.0.smithi083.stdout: Close 619570 0.001 0.039 2024-05-08T20:33:36.820 INFO:tasks.workunit.client.0.smithi083.stdout: Rename 35726 0.088 86.062 2024-05-08T20:33:36.820 INFO:tasks.workunit.client.0.smithi083.stdout: Unlink 170762 0.149 112.896 2024-05-08T20:33:36.820 INFO:tasks.workunit.client.0.smithi083.stdout: Deltree 20 2.271 5.141 2024-05-08T20:33:36.821 INFO:tasks.workunit.client.0.smithi083.stdout: Mkdir 10 0.003 0.004 2024-05-08T20:33:36.821 INFO:tasks.workunit.client.0.smithi083.stdout: Qpathinfo 764788 0.015 91.451 2024-05-08T20:33:36.821 INFO:tasks.workunit.client.0.smithi083.stdout: Qfileinfo 133843 0.001 0.023 2024-05-08T20:33:36.821 INFO:tasks.workunit.client.0.smithi083.stdout: Qfsinfo 140443 0.002 0.275 2024-05-08T20:33:36.821 INFO:tasks.workunit.client.0.smithi083.stdout: Sfileinfo 68610 0.365 98.146 2024-05-08T20:33:36.821 INFO:tasks.workunit.client.0.smithi083.stdout: Find 295807 0.019 2.280 2024-05-08T20:33:36.821 INFO:tasks.workunit.client.0.smithi083.stdout: WriteX 420270 0.113 184.064 2024-05-08T20:33:36.821 INFO:tasks.workunit.client.0.smithi083.stdout: ReadX 1322839 0.007 98.380 2024-05-08T20:33:36.821 INFO:tasks.workunit.client.0.smithi083.stdout: LockX 2742 0.003 0.020 2024-05-08T20:33:36.821 INFO:tasks.workunit.client.0.smithi083.stdout: UnlockX 2742 0.002 0.022 2024-05-08T20:33:36.821 INFO:tasks.workunit.client.0.smithi083.stdout: Flush 59192 98.820 957.938 2024-05-08T20:33:36.821 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-05-08T20:33:36.821 INFO:tasks.workunit.client.0.smithi083.stdout:Throughput 44.2017 MB/sec 10 clients 10 procs max_latency=957.944 ms 2024-05-08T20:33:36.822 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-08T20:33:36.822 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-08T20:33:36.896 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-05-08T20:33:36.896 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-08T20:33:36.988 INFO:tasks.workunit.client.1.smithi188.stdout: 0 cleanup 601 sec 2024-05-08T20:33:36.988 INFO:tasks.workunit.client.1.smithi188.stdout: 2024-05-08T20:33:36.988 INFO:tasks.workunit.client.1.smithi188.stdout: Operation Count AvgLat MaxLat 2024-05-08T20:33:36.988 INFO:tasks.workunit.client.1.smithi188.stdout: ---------------------------------------- 2024-05-08T20:33:36.989 INFO:tasks.workunit.client.1.smithi188.stdout: NTCreateX 424766 4.176 153.392 2024-05-08T20:33:36.989 INFO:tasks.workunit.client.1.smithi188.stdout: Close 311711 0.060 10.209 2024-05-08T20:33:36.989 INFO:tasks.workunit.client.1.smithi188.stdout: Rename 18031 11.202 144.046 2024-05-08T20:33:36.989 INFO:tasks.workunit.client.1.smithi188.stdout: Unlink 86006 2.796 37.864 2024-05-08T20:33:36.989 INFO:tasks.workunit.client.1.smithi188.stdout: Qpathinfo 385108 2.545 156.305 2024-05-08T20:33:36.989 INFO:tasks.workunit.client.1.smithi188.stdout: Qfileinfo 67150 0.134 15.317 2024-05-08T20:33:36.989 INFO:tasks.workunit.client.1.smithi188.stdout: Qfsinfo 70781 1.056 42.084 2024-05-08T20:33:36.989 INFO:tasks.workunit.client.1.smithi188.stdout: Sfileinfo 34630 0.878 20.090 2024-05-08T20:33:36.989 INFO:tasks.workunit.client.1.smithi188.stdout: Find 148949 14.425 180.604 2024-05-08T20:33:36.989 INFO:tasks.workunit.client.1.smithi188.stdout: WriteX 209919 0.277 137.007 2024-05-08T20:33:36.989 INFO:tasks.workunit.client.1.smithi188.stdout: ReadX 667408 0.005 5.664 2024-05-08T20:33:36.989 INFO:tasks.workunit.client.1.smithi188.stdout: LockX 1390 0.969 23.190 2024-05-08T20:33:36.989 INFO:tasks.workunit.client.1.smithi188.stdout: UnlockX 1390 0.664 19.932 2024-05-08T20:33:36.989 INFO:tasks.workunit.client.1.smithi188.stdout: Flush 29796 15.160 157.897 2024-05-08T20:33:36.990 INFO:tasks.workunit.client.1.smithi188.stdout: 2024-05-08T20:33:36.990 INFO:tasks.workunit.client.1.smithi188.stdout:Throughput 22.1009 MB/sec 10 clients 10 procs max_latency=180.607 ms 2024-05-08T20:33:36.990 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-08T20:33:36.990 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-05-08T20:33:37.226 DEBUG:teuthology.parallel:result is None 2024-05-08T20:33:37.249 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.1... 2024-05-08T20:33:37.249 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-05-08T20:33:37.595 DEBUG:teuthology.parallel:result is None 2024-05-08T20:33:37.595 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-05-08T20:33:37.622 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-05-08T20:33:37.623 DEBUG:teuthology.orchestra.run.smithi083:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-05-08T20:33:37.680 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-05-08T20:33:37.681 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-05-08T20:33:37.715 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-05-08T20:33:37.715 INFO:teuthology.run_tasks:Running task pexec... 2024-05-08T20:33:37.727 INFO:teuthology.task.pexec:Executing custom commands... 2024-05-08T20:33:37.728 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi083.front.sepia.ceph.com 2024-05-08T20:33:37.728 DEBUG:teuthology.orchestra.run.smithi083:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-05-08T20:33:37.728 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi188.front.sepia.ceph.com 2024-05-08T20:33:37.728 DEBUG:teuthology.orchestra.run.smithi188:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-05-08T20:33:38.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:38 smithi083 ceph-mon[38695]: pgmap v1026: 129 pgs: 129 active+clean; 420 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 17 KiB/s rd, 9.6 MiB/s wr, 480 op/s 2024-05-08T20:33:38.535 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:38:27 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T20:33:38.564 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:38:38 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T20:33:38.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:38 smithi188 ceph-mon[48761]: pgmap v1026: 129 pgs: 129 active+clean; 420 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 17 KiB/s rd, 9.6 MiB/s wr, 480 op/s 2024-05-08T20:33:39.302 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:39 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:33:39.601 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:39 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:33:40.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:40 smithi188 ceph-mon[48761]: pgmap v1027: 129 pgs: 129 active+clean; 417 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 401 op/s 2024-05-08T20:33:40.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:40 smithi083 ceph-mon[38695]: pgmap v1027: 129 pgs: 129 active+clean; 417 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 401 op/s 2024-05-08T20:33:40.972 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T20:33:40.973 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T20:33:40.973 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-05-08T20:33:40.973 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T20:33:40.973 INFO:teuthology.orchestra.run.smithi083.stdout:Resetting modules: 2024-05-08T20:33:40.973 INFO:teuthology.orchestra.run.smithi083.stdout: container-tools 2024-05-08T20:33:40.974 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:33:40.974 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-05-08T20:33:40.974 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T20:33:40.974 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:33:41.021 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T20:33:41.022 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T20:33:41.022 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-05-08T20:33:41.022 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T20:33:41.022 INFO:teuthology.orchestra.run.smithi188.stdout:Resetting modules: 2024-05-08T20:33:41.022 INFO:teuthology.orchestra.run.smithi188.stdout: container-tools 2024-05-08T20:33:41.023 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T20:33:41.023 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-05-08T20:33:41.023 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T20:33:41.023 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T20:33:41.269 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T20:33:41.284 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T20:33:42.119 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:38:31 ago on Wed 08 May 2024 07:55:11 PM UTC. 2024-05-08T20:33:42.121 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:38:42 ago on Wed 08 May 2024 07:55:00 PM UTC. 2024-05-08T20:33:42.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:42 smithi188 ceph-mon[48761]: pgmap v1028: 129 pgs: 129 active+clean; 319 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 23 KiB/s rd, 8.9 MiB/s wr, 482 op/s 2024-05-08T20:33:42.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:42 smithi083 ceph-mon[38695]: pgmap v1028: 129 pgs: 129 active+clean; 319 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 23 KiB/s rd, 8.9 MiB/s wr, 482 op/s 2024-05-08T20:33:44.289 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-05-08T20:33:44.318 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-05-08T20:33:44.330 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T20:33:44.330 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-05-08T20:33:44.331 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T20:33:44.331 INFO:teuthology.orchestra.run.smithi083.stdout:Installing group/module packages: 2024-05-08T20:33:44.331 INFO:teuthology.orchestra.run.smithi083.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-05-08T20:33:44.331 INFO:teuthology.orchestra.run.smithi083.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-05-08T20:33:44.331 INFO:teuthology.orchestra.run.smithi083.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-05-08T20:33:44.331 INFO:teuthology.orchestra.run.smithi083.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-05-08T20:33:44.331 INFO:teuthology.orchestra.run.smithi083.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-05-08T20:33:44.331 INFO:teuthology.orchestra.run.smithi083.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-05-08T20:33:44.331 INFO:teuthology.orchestra.run.smithi083.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-05-08T20:33:44.331 INFO:teuthology.orchestra.run.smithi083.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-05-08T20:33:44.331 INFO:teuthology.orchestra.run.smithi083.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-05-08T20:33:44.331 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-05-08T20:33:44.331 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-05-08T20:33:44.332 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-05-08T20:33:44.332 INFO:teuthology.orchestra.run.smithi083.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-05-08T20:33:44.332 INFO:teuthology.orchestra.run.smithi083.stdout:Installing weak dependencies: 2024-05-08T20:33:44.332 INFO:teuthology.orchestra.run.smithi083.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-05-08T20:33:44.332 INFO:teuthology.orchestra.run.smithi083.stdout:Installing module profiles: 2024-05-08T20:33:44.332 INFO:teuthology.orchestra.run.smithi083.stdout: container-tools/common 2024-05-08T20:33:44.332 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:33:44.332 INFO:teuthology.orchestra.run.smithi083.stdout:Enabling module streams: 2024-05-08T20:33:44.332 INFO:teuthology.orchestra.run.smithi083.stdout: container-tools rhel8 2024-05-08T20:33:44.332 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:33:44.332 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-05-08T20:33:44.332 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-05-08T20:33:44.332 INFO:teuthology.orchestra.run.smithi083.stdout:Install 13 Packages 2024-05-08T20:33:44.332 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:33:44.333 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 27 M 2024-05-08T20:33:44.333 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 91 M 2024-05-08T20:33:44.334 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-05-08T20:33:44.359 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T20:33:44.359 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-05-08T20:33:44.359 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T20:33:44.359 INFO:teuthology.orchestra.run.smithi188.stdout:Installing group/module packages: 2024-05-08T20:33:44.359 INFO:teuthology.orchestra.run.smithi188.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-05-08T20:33:44.359 INFO:teuthology.orchestra.run.smithi188.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-05-08T20:33:44.359 INFO:teuthology.orchestra.run.smithi188.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-05-08T20:33:44.359 INFO:teuthology.orchestra.run.smithi188.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-05-08T20:33:44.359 INFO:teuthology.orchestra.run.smithi188.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-05-08T20:33:44.359 INFO:teuthology.orchestra.run.smithi188.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-05-08T20:33:44.359 INFO:teuthology.orchestra.run.smithi188.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-05-08T20:33:44.360 INFO:teuthology.orchestra.run.smithi188.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-05-08T20:33:44.360 INFO:teuthology.orchestra.run.smithi188.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-05-08T20:33:44.360 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-05-08T20:33:44.360 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-05-08T20:33:44.360 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-05-08T20:33:44.360 INFO:teuthology.orchestra.run.smithi188.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-05-08T20:33:44.360 INFO:teuthology.orchestra.run.smithi188.stdout:Installing weak dependencies: 2024-05-08T20:33:44.360 INFO:teuthology.orchestra.run.smithi188.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-05-08T20:33:44.360 INFO:teuthology.orchestra.run.smithi188.stdout:Installing module profiles: 2024-05-08T20:33:44.360 INFO:teuthology.orchestra.run.smithi188.stdout: container-tools/common 2024-05-08T20:33:44.360 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T20:33:44.361 INFO:teuthology.orchestra.run.smithi188.stdout:Enabling module streams: 2024-05-08T20:33:44.361 INFO:teuthology.orchestra.run.smithi188.stdout: container-tools rhel8 2024-05-08T20:33:44.361 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T20:33:44.361 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-05-08T20:33:44.361 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-05-08T20:33:44.361 INFO:teuthology.orchestra.run.smithi188.stdout:Install 13 Packages 2024-05-08T20:33:44.361 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T20:33:44.362 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 27 M 2024-05-08T20:33:44.362 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 91 M 2024-05-08T20:33:44.362 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-05-08T20:33:44.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:44 smithi083 ceph-mon[38695]: pgmap v1029: 129 pgs: 129 active+clean; 304 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 17 KiB/s rd, 5.8 MiB/s wr, 363 op/s 2024-05-08T20:33:44.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:44 smithi188 ceph-mon[48761]: pgmap v1029: 129 pgs: 129 active+clean; 304 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 17 KiB/s rd, 5.8 MiB/s wr, 363 op/s 2024-05-08T20:33:44.674 INFO:teuthology.orchestra.run.smithi083.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 2.4 MB/s | 689 kB 00:00 2024-05-08T20:33:44.716 INFO:teuthology.orchestra.run.smithi083.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 3.1 MB/s | 1.0 MB 00:00 2024-05-08T20:33:44.740 INFO:teuthology.orchestra.run.smithi188.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 2.1 MB/s | 689 kB 00:00 2024-05-08T20:33:44.799 INFO:teuthology.orchestra.run.smithi188.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 2.7 MB/s | 1.0 MB 00:00 2024-05-08T20:33:44.898 INFO:teuthology.orchestra.run.smithi083.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 284 kB/s | 38 kB 00:00 2024-05-08T20:33:44.898 INFO:teuthology.orchestra.run.smithi188.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 253 kB/s | 38 kB 00:00 2024-05-08T20:33:44.900 INFO:teuthology.orchestra.run.smithi083.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 1.4 MB/s | 258 kB 00:00 2024-05-08T20:33:44.982 INFO:teuthology.orchestra.run.smithi188.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 1.4 MB/s | 258 kB 00:00 2024-05-08T20:33:45.059 INFO:teuthology.orchestra.run.smithi083.stdout:(5/13): buildah-1.33.6-2.module_el8+976+1136073 15 MB/s | 9.6 MB 00:00 2024-05-08T20:33:45.100 INFO:teuthology.orchestra.run.smithi083.stdout:(6/13): python3-podman-4.9.0-1.module_el8+847+7 900 kB/s | 180 kB 00:00 2024-05-08T20:33:45.149 INFO:teuthology.orchestra.run.smithi188.stdout:(5/13): buildah-1.33.6-2.module_el8+976+1136073 13 MB/s | 9.6 MB 00:00 2024-05-08T20:33:45.176 INFO:teuthology.orchestra.run.smithi083.stdout:(7/13): python3-pytoml-0.1.14-5.git7dea353.el8. 212 kB/s | 25 kB 00:00 2024-05-08T20:33:45.210 INFO:teuthology.orchestra.run.smithi188.stdout:(6/13): python3-podman-4.9.0-1.module_el8+847+7 791 kB/s | 180 kB 00:00 2024-05-08T20:33:45.284 INFO:teuthology.orchestra.run.smithi083.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 8.6 MB/s | 4.1 MB 00:00 2024-05-08T20:33:45.291 INFO:teuthology.orchestra.run.smithi188.stdout:(7/13): python3-pytoml-0.1.14-5.git7dea353.el8. 174 kB/s | 25 kB 00:00 2024-05-08T20:33:45.310 INFO:teuthology.orchestra.run.smithi083.stdout:(9/13): python3-pyxdg-0.25-16.el8.noarch.rpm 449 kB/s | 94 kB 00:00 2024-05-08T20:33:45.375 INFO:teuthology.orchestra.run.smithi188.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 8.5 MB/s | 4.1 MB 00:00 2024-05-08T20:33:45.401 INFO:teuthology.orchestra.run.smithi188.stdout:(9/13): python3-pyxdg-0.25-16.el8.noarch.rpm 495 kB/s | 94 kB 00:00 2024-05-08T20:33:45.418 INFO:teuthology.orchestra.run.smithi083.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 486 kB/s | 53 kB 00:00 2024-05-08T20:33:45.517 INFO:teuthology.orchestra.run.smithi188.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 451 kB/s | 53 kB 00:00 2024-05-08T20:33:45.544 INFO:teuthology.orchestra.run.smithi083.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 327 kB/s | 41 kB 00:00 2024-05-08T20:33:45.634 INFO:teuthology.orchestra.run.smithi188.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 351 kB/s | 41 kB 00:00 2024-05-08T20:33:45.652 INFO:teuthology.orchestra.run.smithi083.stdout:(12/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 6.9 MB/s | 2.5 MB 00:00 2024-05-08T20:33:45.768 INFO:teuthology.orchestra.run.smithi188.stdout:(12/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 6.4 MB/s | 2.5 MB 00:00 2024-05-08T20:33:45.820 INFO:teuthology.orchestra.run.smithi083.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 14 MB/s | 8.8 MB 00:00 2024-05-08T20:33:45.820 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-05-08T20:33:45.820 INFO:teuthology.orchestra.run.smithi083.stdout:Total 18 MB/s | 27 MB 00:01 2024-05-08T20:33:45.902 INFO:teuthology.orchestra.run.smithi188.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 14 MB/s | 8.8 MB 00:00 2024-05-08T20:33:45.902 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-05-08T20:33:45.902 INFO:teuthology.orchestra.run.smithi188.stdout:Total 18 MB/s | 27 MB 00:01 2024-05-08T20:33:46.095 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-05-08T20:33:46.117 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-05-08T20:33:46.117 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-05-08T20:33:46.182 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-05-08T20:33:46.203 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-05-08T20:33:46.203 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-05-08T20:33:46.387 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-05-08T20:33:46.391 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-05-08T20:33:46.452 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-05-08T20:33:46.455 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-05-08T20:33:46.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:46 smithi188 ceph-mon[48761]: pgmap v1030: 129 pgs: 129 active+clean; 219 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 17 KiB/s rd, 5.4 MiB/s wr, 337 op/s 2024-05-08T20:33:46.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:46 smithi083 ceph-mon[38695]: pgmap v1030: 129 pgs: 129 active+clean; 219 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 17 KiB/s rd, 5.4 MiB/s wr, 337 op/s 2024-05-08T20:33:47.006 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-05-08T20:33:47.124 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-05-08T20:33:47.964 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-05-08T20:33:48.076 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-05-08T20:33:48.226 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-05-08T20:33:48.306 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-05-08T20:33:48.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:48 smithi083 ceph-mon[38695]: pgmap v1031: 129 pgs: 129 active+clean; 199 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 17 KiB/s rd, 3.8 MiB/s wr, 311 op/s 2024-05-08T20:33:48.537 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-05-08T20:33:48.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:48 smithi188 ceph-mon[48761]: pgmap v1031: 129 pgs: 129 active+clean; 199 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 17 KiB/s rd, 3.8 MiB/s wr, 311 op/s 2024-05-08T20:33:48.779 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-05-08T20:33:48.818 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-05-08T20:33:49.037 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-05-08T20:33:49.144 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-05-08T20:33:49.449 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-05-08T20:33:49.599 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-05-08T20:33:50.024 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-05-08T20:33:50.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:50 smithi188 ceph-mon[48761]: pgmap v1032: 129 pgs: 129 active+clean; 198 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 11 KiB/s rd, 1.6 MiB/s wr, 158 op/s 2024-05-08T20:33:50.601 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-05-08T20:33:50.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:50 smithi083 ceph-mon[38695]: pgmap v1032: 129 pgs: 129 active+clean; 198 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 11 KiB/s rd, 1.6 MiB/s wr, 158 op/s 2024-05-08T20:33:50.839 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-05-08T20:33:51.011 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-05-08T20:33:51.118 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-05-08T20:33:51.251 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-05-08T20:33:51.489 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-05-08T20:33:51.634 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-05-08T20:33:51.906 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-05-08T20:33:51.989 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-05-08T20:33:52.152 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-05-08T20:33:52.226 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-05-08T20:33:52.427 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-05-08T20:33:52.468 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-05-08T20:33:52.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:52 smithi188 ceph-mon[48761]: pgmap v1033: 129 pgs: 129 active+clean; 198 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 1.3 MiB/s wr, 144 op/s 2024-05-08T20:33:52.671 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-05-08T20:33:52.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:52 smithi083 ceph-mon[38695]: pgmap v1033: 129 pgs: 129 active+clean; 198 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 12 KiB/s rd, 1.3 MiB/s wr, 144 op/s 2024-05-08T20:33:52.888 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-05-08T20:33:52.888 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-05-08T20:33:52.888 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-05-08T20:33:52.888 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-05-08T20:33:52.888 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-05-08T20:33:52.888 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-05-08T20:33:52.888 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-05-08T20:33:52.888 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-05-08T20:33:52.888 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-05-08T20:33:52.888 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-05-08T20:33:52.888 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-05-08T20:33:52.889 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-05-08T20:33:52.889 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-05-08T20:33:53.162 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-05-08T20:33:53.162 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-05-08T20:33:53.162 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-05-08T20:33:53.162 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-05-08T20:33:53.162 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-05-08T20:33:53.162 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-05-08T20:33:53.162 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-05-08T20:33:53.162 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-05-08T20:33:53.162 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-05-08T20:33:53.162 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-05-08T20:33:53.162 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-05-08T20:33:53.163 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-05-08T20:33:53.163 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-05-08T20:33:53.451 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-05-08T20:33:53.451 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T20:33:53.451 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-05-08T20:33:53.451 INFO:teuthology.orchestra.run.smithi188.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-05-08T20:33:53.451 INFO:teuthology.orchestra.run.smithi188.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-05-08T20:33:53.451 INFO:teuthology.orchestra.run.smithi188.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-05-08T20:33:53.451 INFO:teuthology.orchestra.run.smithi188.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-05-08T20:33:53.451 INFO:teuthology.orchestra.run.smithi188.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-05-08T20:33:53.451 INFO:teuthology.orchestra.run.smithi188.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-05-08T20:33:53.451 INFO:teuthology.orchestra.run.smithi188.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-05-08T20:33:53.451 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-05-08T20:33:53.451 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-05-08T20:33:53.451 INFO:teuthology.orchestra.run.smithi188.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-05-08T20:33:53.452 INFO:teuthology.orchestra.run.smithi188.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-05-08T20:33:53.452 INFO:teuthology.orchestra.run.smithi188.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-05-08T20:33:53.452 INFO:teuthology.orchestra.run.smithi188.stdout: yajl-2.1.0-12.el8.x86_64 2024-05-08T20:33:53.452 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-05-08T20:33:53.452 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-05-08T20:33:53.724 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-05-08T20:33:53.725 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:33:53.725 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-05-08T20:33:53.725 INFO:teuthology.orchestra.run.smithi083.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-05-08T20:33:53.725 INFO:teuthology.orchestra.run.smithi083.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-05-08T20:33:53.725 INFO:teuthology.orchestra.run.smithi083.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-05-08T20:33:53.725 INFO:teuthology.orchestra.run.smithi083.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-05-08T20:33:53.725 INFO:teuthology.orchestra.run.smithi083.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-05-08T20:33:53.725 INFO:teuthology.orchestra.run.smithi083.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-05-08T20:33:53.725 INFO:teuthology.orchestra.run.smithi083.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-05-08T20:33:53.725 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-05-08T20:33:53.725 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-05-08T20:33:53.725 INFO:teuthology.orchestra.run.smithi083.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-05-08T20:33:53.725 INFO:teuthology.orchestra.run.smithi083.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-05-08T20:33:53.725 INFO:teuthology.orchestra.run.smithi083.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-05-08T20:33:53.726 INFO:teuthology.orchestra.run.smithi083.stdout: yajl-2.1.0-12.el8.x86_64 2024-05-08T20:33:53.726 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:33:53.726 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-05-08T20:33:53.785 DEBUG:teuthology.parallel:result is None 2024-05-08T20:33:54.068 DEBUG:teuthology.parallel:result is None 2024-05-08T20:33:54.069 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-05-08T20:33:54.079 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-05-08T20:33:54.079 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-05-08T20:33:54.079 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T20:33:54.100 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-05-08T20:33:54.100 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T20:33:54.159 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T20:33:54.498 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:33:54.499 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":72,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","created":"2024-05-08T19:59:07.152446+0000","modified":"2024-05-08T20:08:24.896341+0000","last_up_change":"2024-05-08T20:05:57.759893+0000","last_in_change":"2024-05-08T20:05:40.402230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T20:03:44.088148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.foo.meta","create_time":"2024-05-08T20:07:27.766436+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.data","create_time":"2024-05-08T20:07:28.151095+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.bar.meta","create_time":"2024-05-08T20:07:34.236025+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bar"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs.bar.data","create_time":"2024-05-08T20:07:35.187905+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bar"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78051d3d-3644-4876-af5f-8c956a8491bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6803","nonce":345419579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6805","nonce":345419579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6809","nonce":345419579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6807","nonce":345419579}]},"public_addr":"172.21.15.83:6803/345419579","cluster_addr":"172.21.15.83:6805/345419579","heartbeat_back_addr":"172.21.15.83:6809/345419579","heartbeat_front_addr":"172.21.15.83:6807/345419579","state":["exists","up"]},{"osd":1,"uuid":"bc62b8e9-e30e-41f1-aaee-4c4e356be487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6811","nonce":1992633959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6813","nonce":1992633959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6817","nonce":1992633959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6815","nonce":1992633959}]},"public_addr":"172.21.15.83:6811/1992633959","cluster_addr":"172.21.15.83:6813/1992633959","heartbeat_back_addr":"172.21.15.83:6817/1992633959","heartbeat_front_addr":"172.21.15.83:6815/1992633959","state":["exists","up"]},{"osd":2,"uuid":"4ae0453a-718b-445f-ad00-a3696fec1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6819","nonce":561864588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6821","nonce":561864588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6825","nonce":561864588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6823","nonce":561864588}]},"public_addr":"172.21.15.83:6819/561864588","cluster_addr":"172.21.15.83:6821/561864588","heartbeat_back_addr":"172.21.15.83:6825/561864588","heartbeat_front_addr":"172.21.15.83:6823/561864588","state":["exists","up"]},{"osd":3,"uuid":"2085ec85-187a-4d69-a721-793cfed7d044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6827","nonce":866184273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6829","nonce":866184273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6833","nonce":866184273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6831","nonce":866184273}]},"public_addr":"172.21.15.83:6827/866184273","cluster_addr":"172.21.15.83:6829/866184273","heartbeat_back_addr":"172.21.15.83:6833/866184273","heartbeat_front_addr":"172.21.15.83:6831/866184273","state":["exists","up"]},{"osd":4,"uuid":"b92dad8b-5bd0-4cd1-ba8d-2920927f132e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6801","nonce":422247785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6803","nonce":422247785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6807","nonce":422247785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6805","nonce":422247785}]},"public_addr":"172.21.15.188:6801/422247785","cluster_addr":"172.21.15.188:6803/422247785","heartbeat_back_addr":"172.21.15.188:6807/422247785","heartbeat_front_addr":"172.21.15.188:6805/422247785","state":["exists","up"]},{"osd":5,"uuid":"aeeda050-668f-47a2-90d0-32e1aa35bfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6809","nonce":841652631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6811","nonce":841652631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6815","nonce":841652631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6813","nonce":841652631}]},"public_addr":"172.21.15.188:6809/841652631","cluster_addr":"172.21.15.188:6811/841652631","heartbeat_back_addr":"172.21.15.188:6815/841652631","heartbeat_front_addr":"172.21.15.188:6813/841652631","state":["exists","up"]},{"osd":6,"uuid":"d3720c87-ec2b-490a-937b-a29833f0267c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6817","nonce":118382910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6819","nonce":118382910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6823","nonce":118382910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6821","nonce":118382910}]},"public_addr":"172.21.15.188:6817/118382910","cluster_addr":"172.21.15.188:6819/118382910","heartbeat_back_addr":"172.21.15.188:6823/118382910","heartbeat_front_addr":"172.21.15.188:6821/118382910","state":["exists","up"]},{"osd":7,"uuid":"8fffd2a6-163d-4a9b-adc2-25329a3803ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6825","nonce":3349446159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6827","nonce":3349446159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6831","nonce":3349446159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6829","nonce":3349446159}]},"public_addr":"172.21.15.188:6825/3349446159","cluster_addr":"172.21.15.188:6827/3349446159","heartbeat_back_addr":"172.21.15.188:6831/3349446159","heartbeat_front_addr":"172.21.15.188:6829/3349446159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:02:42.828169+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:11.944380+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:41.150631+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:11.648151+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:36.764713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:03.134028+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:29.964086+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:56.788654+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":7,"to":4}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.188:6833/1664076600":"2024-05-09T20:07:44.002323+0000","172.21.15.83:0/3512048858":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6800/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/2443488799":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6801/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/2142540104":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/890224504":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6801/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/2981330630":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6800/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6800/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3109058932":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3117466271":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/1765857891":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/3612788753":"2024-05-09T20:01:23.760420+0000","172.21.15.83:6801/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.188:6832/1664076600":"2024-05-09T20:07:44.002323+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T20:33:54.513 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T20:03:44.088148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.foo.meta', 'create_time': '2024-05-08T20:07:27.766436+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.foo.data', 'create_time': '2024-05-08T20:07:28.151095+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs.bar.meta', 'create_time': '2024-05-08T20:07:34.236025+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'bar'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs.bar.data', 'create_time': '2024-05-08T20:07:35.187905+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'bar'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T20:33:54.514 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T20:33:54.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:54 smithi188 ceph-mon[48761]: pgmap v1034: 129 pgs: 129 active+clean; 198 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 341 B/s rd, 39 KiB/s wr, 39 op/s 2024-05-08T20:33:54.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:54 smithi188 ceph-mon[48761]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:33:54.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:54 smithi083 ceph-mon[38695]: pgmap v1034: 129 pgs: 129 active+clean; 198 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 341 B/s rd, 39 KiB/s wr, 39 op/s 2024-05-08T20:33:54.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:54 smithi083 ceph-mon[38695]: from='mgr.14211 172.21.15.83:0/2971314542' entity='mgr.smithi083.fgcgxm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-08T20:33:54.846 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-05-08T20:33:54.860 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.meta pg_num 2024-05-08T20:33:55.182 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:33:55.197 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.data pg_num 2024-05-08T20:33:55.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:55 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/761529469' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:33:55.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:55 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/4053080035' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-08T20:33:55.447 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:55 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3730959961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:33:55.525 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:33:55.540 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.meta pg_num 2024-05-08T20:33:55.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:55 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/761529469' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:33:55.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:55 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/4053080035' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-08T20:33:55.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:55 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/3730959961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:33:55.867 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:33:55.881 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.data pg_num 2024-05-08T20:33:56.208 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:33:56.223 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-08T20:33:56.562 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.188:6833/1664076600 2024-05-09T20:07:44.002323+0000 2024-05-08T20:33:56.563 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/3512048858 2024-05-09T19:59:36.209291+0000 2024-05-08T20:33:56.563 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6800/792726797 2024-05-09T20:00:11.913247+0000 2024-05-08T20:33:56.563 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/2443488799 2024-05-09T20:00:11.913247+0000 2024-05-08T20:33:56.563 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6801/792726797 2024-05-09T20:00:11.913247+0000 2024-05-08T20:33:56.563 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/2142540104 2024-05-09T20:01:23.760420+0000 2024-05-08T20:33:56.563 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/890224504 2024-05-09T19:59:36.209291+0000 2024-05-08T20:33:56.563 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6801/3694300604 2024-05-09T19:59:36.209291+0000 2024-05-08T20:33:56.563 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/2981330630 2024-05-09T19:59:36.209291+0000 2024-05-08T20:33:56.563 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6800/3694300604 2024-05-09T19:59:36.209291+0000 2024-05-08T20:33:56.563 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6800/3762483871 2024-05-09T20:01:23.760420+0000 2024-05-08T20:33:56.563 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/3109058932 2024-05-09T20:01:23.760420+0000 2024-05-08T20:33:56.563 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/3117466271 2024-05-09T20:00:11.913247+0000 2024-05-08T20:33:56.563 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/1765857891 2024-05-09T20:00:11.913247+0000 2024-05-08T20:33:56.564 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/3612788753 2024-05-09T20:01:23.760420+0000 2024-05-08T20:33:56.564 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6801/3762483871 2024-05-09T20:01:23.760420+0000 2024-05-08T20:33:56.564 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.188:6832/1664076600 2024-05-09T20:07:44.002323+0000 2024-05-08T20:33:56.565 INFO:teuthology.orchestra.run.smithi083.stderr:listed 17 entries 2024-05-08T20:33:56.576 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:56 smithi083 ceph-mon[38695]: pgmap v1035: 129 pgs: 129 active+clean; 198 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 341 B/s rd, 39 KiB/s wr, 33 op/s 2024-05-08T20:33:56.576 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:56 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2632413532' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-05-08T20:33:56.576 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:56 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/419540960' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:33:56.576 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:56 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/221085368' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-05-08T20:33:56.577 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-05-08T20:33:56.577 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T20:33:56.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:56 smithi188 ceph-mon[48761]: pgmap v1035: 129 pgs: 129 active+clean; 198 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 341 B/s rd, 39 KiB/s wr, 33 op/s 2024-05-08T20:33:56.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:56 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/2632413532' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-05-08T20:33:56.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:56 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/419540960' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:33:56.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:56 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/221085368' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-05-08T20:33:56.601 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T20:33:56.923 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-05-08T20:33:56.923 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":72,"fsid":"526038a6-0d75-11ef-bc97-c7b262605968","created":"2024-05-08T19:59:07.152446+0000","modified":"2024-05-08T20:08:24.896341+0000","last_up_change":"2024-05-08T20:05:57.759893+0000","last_in_change":"2024-05-08T20:05:40.402230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T20:03:44.088148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.foo.meta","create_time":"2024-05-08T20:07:27.766436+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.data","create_time":"2024-05-08T20:07:28.151095+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.bar.meta","create_time":"2024-05-08T20:07:34.236025+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bar"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs.bar.data","create_time":"2024-05-08T20:07:35.187905+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bar"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78051d3d-3644-4876-af5f-8c956a8491bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6803","nonce":345419579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6805","nonce":345419579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6809","nonce":345419579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":345419579},{"type":"v1","addr":"172.21.15.83:6807","nonce":345419579}]},"public_addr":"172.21.15.83:6803/345419579","cluster_addr":"172.21.15.83:6805/345419579","heartbeat_back_addr":"172.21.15.83:6809/345419579","heartbeat_front_addr":"172.21.15.83:6807/345419579","state":["exists","up"]},{"osd":1,"uuid":"bc62b8e9-e30e-41f1-aaee-4c4e356be487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6811","nonce":1992633959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6813","nonce":1992633959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6817","nonce":1992633959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1992633959},{"type":"v1","addr":"172.21.15.83:6815","nonce":1992633959}]},"public_addr":"172.21.15.83:6811/1992633959","cluster_addr":"172.21.15.83:6813/1992633959","heartbeat_back_addr":"172.21.15.83:6817/1992633959","heartbeat_front_addr":"172.21.15.83:6815/1992633959","state":["exists","up"]},{"osd":2,"uuid":"4ae0453a-718b-445f-ad00-a3696fec1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6819","nonce":561864588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6821","nonce":561864588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6825","nonce":561864588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":561864588},{"type":"v1","addr":"172.21.15.83:6823","nonce":561864588}]},"public_addr":"172.21.15.83:6819/561864588","cluster_addr":"172.21.15.83:6821/561864588","heartbeat_back_addr":"172.21.15.83:6825/561864588","heartbeat_front_addr":"172.21.15.83:6823/561864588","state":["exists","up"]},{"osd":3,"uuid":"2085ec85-187a-4d69-a721-793cfed7d044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6827","nonce":866184273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6829","nonce":866184273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6833","nonce":866184273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":866184273},{"type":"v1","addr":"172.21.15.83:6831","nonce":866184273}]},"public_addr":"172.21.15.83:6827/866184273","cluster_addr":"172.21.15.83:6829/866184273","heartbeat_back_addr":"172.21.15.83:6833/866184273","heartbeat_front_addr":"172.21.15.83:6831/866184273","state":["exists","up"]},{"osd":4,"uuid":"b92dad8b-5bd0-4cd1-ba8d-2920927f132e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6801","nonce":422247785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6803","nonce":422247785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6807","nonce":422247785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":422247785},{"type":"v1","addr":"172.21.15.188:6805","nonce":422247785}]},"public_addr":"172.21.15.188:6801/422247785","cluster_addr":"172.21.15.188:6803/422247785","heartbeat_back_addr":"172.21.15.188:6807/422247785","heartbeat_front_addr":"172.21.15.188:6805/422247785","state":["exists","up"]},{"osd":5,"uuid":"aeeda050-668f-47a2-90d0-32e1aa35bfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6809","nonce":841652631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6811","nonce":841652631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6815","nonce":841652631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":841652631},{"type":"v1","addr":"172.21.15.188:6813","nonce":841652631}]},"public_addr":"172.21.15.188:6809/841652631","cluster_addr":"172.21.15.188:6811/841652631","heartbeat_back_addr":"172.21.15.188:6815/841652631","heartbeat_front_addr":"172.21.15.188:6813/841652631","state":["exists","up"]},{"osd":6,"uuid":"d3720c87-ec2b-490a-937b-a29833f0267c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6817","nonce":118382910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6819","nonce":118382910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6823","nonce":118382910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":118382910},{"type":"v1","addr":"172.21.15.188:6821","nonce":118382910}]},"public_addr":"172.21.15.188:6817/118382910","cluster_addr":"172.21.15.188:6819/118382910","heartbeat_back_addr":"172.21.15.188:6823/118382910","heartbeat_front_addr":"172.21.15.188:6821/118382910","state":["exists","up"]},{"osd":7,"uuid":"8fffd2a6-163d-4a9b-adc2-25329a3803ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6825","nonce":3349446159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6827","nonce":3349446159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6831","nonce":3349446159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":3349446159},{"type":"v1","addr":"172.21.15.188:6829","nonce":3349446159}]},"public_addr":"172.21.15.188:6825/3349446159","cluster_addr":"172.21.15.188:6827/3349446159","heartbeat_back_addr":"172.21.15.188:6831/3349446159","heartbeat_front_addr":"172.21.15.188:6829/3349446159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:02:42.828169+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:11.944380+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:03:41.150631+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:11.648151+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:04:36.764713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:03.134028+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:29.964086+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T20:05:56.788654+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":7,"to":4}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.188:6833/1664076600":"2024-05-09T20:07:44.002323+0000","172.21.15.83:0/3512048858":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6800/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/2443488799":"2024-05-09T20:00:11.913247+0000","172.21.15.83:6801/792726797":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/2142540104":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/890224504":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6801/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:0/2981330630":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6800/3694300604":"2024-05-09T19:59:36.209291+0000","172.21.15.83:6800/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3109058932":"2024-05-09T20:01:23.760420+0000","172.21.15.83:0/3117466271":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/1765857891":"2024-05-09T20:00:11.913247+0000","172.21.15.83:0/3612788753":"2024-05-09T20:01:23.760420+0000","172.21.15.83:6801/3762483871":"2024-05-09T20:01:23.760420+0000","172.21.15.188:6832/1664076600":"2024-05-09T20:07:44.002323+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T20:33:56.937 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-08T20:03:44.088148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.foo.meta', 'create_time': '2024-05-08T20:07:27.766436+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.foo.data', 'create_time': '2024-05-08T20:07:28.151095+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs.bar.meta', 'create_time': '2024-05-08T20:07:34.236025+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'bar'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs.bar.data', 'create_time': '2024-05-08T20:07:35.187905+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'bar'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-08T20:33:56.938 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-08T20:33:57.263 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-05-08T20:33:57.277 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.meta pg_num 2024-05-08T20:33:57.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:57 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/1471496797' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-08T20:33:57.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:57 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/3897505633' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:33:57.598 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:57 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/1783476427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-08T20:33:57.599 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:33:57.613 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:57 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/1471496797' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-08T20:33:57.613 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:57 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3897505633' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-08T20:33:57.613 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:57 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/1783476427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-08T20:33:57.614 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.data pg_num 2024-05-08T20:33:57.940 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:33:57.954 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.meta pg_num 2024-05-08T20:33:58.280 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:33:58.295 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.data pg_num 2024-05-08T20:33:58.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:58 smithi188 ceph-mon[48761]: pgmap v1036: 129 pgs: 129 active+clean; 198 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 341 B/s rd, 39 KiB/s wr, 10 op/s 2024-05-08T20:33:58.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:58 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/3748990548' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:33:58.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:58 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/2154888604' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-05-08T20:33:58.598 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:58 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/1916074455' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:33:58.621 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-05-08T20:33:58.634 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:58 smithi083 ceph-mon[38695]: pgmap v1036: 129 pgs: 129 active+clean; 198 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 341 B/s rd, 39 KiB/s wr, 10 op/s 2024-05-08T20:33:58.634 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:58 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/3748990548' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:33:58.635 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:58 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2154888604' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-05-08T20:33:58.635 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:58 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/1916074455' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-05-08T20:33:58.636 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-08T20:33:58.976 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.188:6833/1664076600 2024-05-09T20:07:44.002323+0000 2024-05-08T20:33:58.976 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/3512048858 2024-05-09T19:59:36.209291+0000 2024-05-08T20:33:58.977 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6800/792726797 2024-05-09T20:00:11.913247+0000 2024-05-08T20:33:58.977 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/2443488799 2024-05-09T20:00:11.913247+0000 2024-05-08T20:33:58.977 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6801/792726797 2024-05-09T20:00:11.913247+0000 2024-05-08T20:33:58.977 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/2142540104 2024-05-09T20:01:23.760420+0000 2024-05-08T20:33:58.977 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/890224504 2024-05-09T19:59:36.209291+0000 2024-05-08T20:33:58.977 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6801/3694300604 2024-05-09T19:59:36.209291+0000 2024-05-08T20:33:58.977 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/2981330630 2024-05-09T19:59:36.209291+0000 2024-05-08T20:33:58.977 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6800/3694300604 2024-05-09T19:59:36.209291+0000 2024-05-08T20:33:58.977 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6800/3762483871 2024-05-09T20:01:23.760420+0000 2024-05-08T20:33:58.977 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/3109058932 2024-05-09T20:01:23.760420+0000 2024-05-08T20:33:58.977 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/3117466271 2024-05-09T20:00:11.913247+0000 2024-05-08T20:33:58.977 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/1765857891 2024-05-09T20:00:11.913247+0000 2024-05-08T20:33:58.977 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:0/3612788753 2024-05-09T20:01:23.760420+0000 2024-05-08T20:33:58.977 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6801/3762483871 2024-05-09T20:01:23.760420+0000 2024-05-08T20:33:58.978 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.188:6832/1664076600 2024-05-09T20:07:44.002323+0000 2024-05-08T20:33:58.979 INFO:teuthology.orchestra.run.smithi083.stderr:listed 17 entries 2024-05-08T20:33:58.990 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi188.front.sepia.ceph.com... 2024-05-08T20:33:58.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:33:58.991 DEBUG:teuthology.orchestra.run.smithi188:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-05-08T20:33:59.032 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T20:33:59.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[85148]: fuse finished with error 0 and tester_r 0 2024-05-08T20:33:59.674 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:59 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/2462037301' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-05-08T20:33:59.674 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:33:59 smithi188 ceph-mon[48761]: from='client.? 172.21.15.83:0/2035458415' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-08T20:33:59.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:59 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2462037301' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-05-08T20:33:59.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:33:59 smithi083 ceph-mon[38695]: from='client.? 172.21.15.83:0/2035458415' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-08T20:34:00.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:34:00 smithi083 ceph-mon[38695]: pgmap v1037: 129 pgs: 129 active+clean; 198 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 170 B/s rd, 682 B/s wr, 1 op/s 2024-05-08T20:34:00.847 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:34:00 smithi188 ceph-mon[48761]: pgmap v1037: 129 pgs: 129 active+clean; 198 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 170 B/s rd, 682 B/s wr, 1 op/s 2024-05-08T20:34:02.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:34:02 smithi083 ceph-mon[38695]: pgmap v1038: 129 pgs: 129 active+clean; 198 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 170 B/s rd, 938 B/s wr, 1 op/s 2024-05-08T20:34:02.848 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:34:02 smithi188 ceph-mon[48761]: pgmap v1038: 129 pgs: 129 active+clean; 198 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 170 B/s rd, 938 B/s wr, 1 op/s 2024-05-08T20:34:04.697 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:34:04 smithi083 ceph-mon[38695]: pgmap v1039: 129 pgs: 129 active+clean; 198 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-05-08T20:34:04.726 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:34:04 smithi188 ceph-mon[48761]: pgmap v1039: 129 pgs: 129 active+clean; 198 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-05-08T20:34:05.134 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi188.front.sepia.ceph.com 2024-05-08T20:34:05.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:34:05.135 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-05-08T20:34:05.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:34:05.172 DEBUG:teuthology.orchestra.run.smithi188:> ip netns list 2024-05-08T20:34:05.234 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-05-08T20:34:05.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:34:05.234 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-08T20:34:05.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:34:05.325 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link delete ceph-brx 2024-05-08T20:34:05.407 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-05-08T20:34:05.417 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-05-08T20:34:05.417 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-05-08T20:34:05.418 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-08T20:34:05.441 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251148k,nr_inodes=4062787,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19820 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/f250d2661586c1b977d0610848396d2edec2266d0cee858c1d28cb46d3c5bd6e/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/344W7VQQPULFU5OLHH4H74PRF5:/var/lib/containers/storage/overlay/l/DRR2NEPI4XZI5P5MGC6DNB5IRH,upperdir=/var/lib/containers/storage/overlay/f250d2661586c1b977d0610848396d2edec2266d0cee858c1d28cb46d3c5bd6e/diff,workdir=/var/lib/containers/storage/overlay/f250d2661586c1b977d0610848396d2edec2266d0cee858c1d28cb46d3c5bd6e/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/3dfdf4c41a3e2251754b3c42b3873cbe5cdd1e1ce65eb4e45e5c3b37e1f97a23/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/344W7VQQPULFU5OLHH4H74PRF5:/var/lib/containers/storage/overlay/l/DRR2NEPI4XZI5P5MGC6DNB5IRH,upperdir=/var/lib/containers/storage/overlay/3dfdf4c41a3e2251754b3c42b3873cbe5cdd1e1ce65eb4e45e5c3b37e1f97a23/diff,workdir=/var/lib/containers/storage/overlay/3dfdf4c41a3e2251754b3c42b3873cbe5cdd1e1ce65eb4e45e5c3b37e1f97a23/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0 overlay /var/lib/containers/storage/overlay/4955513be2245fdb831cb966cf7ed487592d590b68c56f9705a9455cb2dd0877/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/344W7VQQPULFU5OLHH4H74PRF5:/var/lib/containers/storage/overlay/l/DRR2NEPI4XZI5P5MGC6DNB5IRH,upperdir=/var/lib/containers/storage/overlay/4955513be2245fdb831cb966cf7ed487592d590b68c56f9705a9455cb2dd0877/diff,workdir=/var/lib/containers/storage/overlay/4955513be2245fdb831cb966cf7ed487592d590b68c56f9705a9455cb2dd0877/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/fb125fe8b015159f4f990018058d5bf7a61ae8800a109ff1b9b8891fc751bf1e/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/344W7VQQPULFU5OLHH4H74PRF5:/var/lib/containers/storage/overlay/l/DRR2NEPI4XZI5P5MGC6DNB5IRH,upperdir=/var/lib/containers/storage/overlay/fb125fe8b015159f4f990018058d5bf7a61ae8800a109ff1b9b8891fc751bf1e/diff,workdir=/var/lib/containers/storage/overlay/fb125fe8b015159f4f990018058d5bf7a61ae8800a109ff1b9b8891fc751bf1e/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/33e0bd8ab656da2ced6be70f664ad6d4107787649ec94849526f0f68ae37527e/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/3CQQNAVZHRZEPD6S5GFUALOWWV:/var/lib/containers/storage/overlay/l/LGWMQ6BAG4FKPXDFPWSZRK3243:/var/lib/containers/storage/overlay/l/L5SZ4MAD66WL34AIKDDQHMTKUN,upperdir=/var/lib/containers/storage/overlay/33e0bd8ab656da2ced6be70f664ad6d4107787649ec94849526f0f68ae37527e/diff,workdir=/var/lib/containers/storage/overlay/33e0bd8ab656da2ced6be70f664ad6d4107787649ec94849526f0f68ae37527e/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/b57f721b7589f6c38cc11d76c7fd232e1d8bf7d0a7e499904c4895c0bd8c94fe/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/O27Z6XSCB64Y6GUU7CIFUZUQ4U:/var/lib/containers/storage/overlay/l/I4EKDTRCJYD2SJMYFPKBJXYL67:/var/lib/containers/storage/overlay/l/BLCZ76EN2K4ZFH6UH6JLA4WLNK:/var/lib/containers/storage/overlay/l/KGJ2OR5WPPQQ5HYI2LL3ANL2XK:/var/lib/containers/storage/overlay/l/XNAEEUZJRZXE2Y4NGMBFSJLUI3:/var/lib/containers/storage/overlay/l/JTBCIKHYPWBMPFPJHIYNILCXPL,upperdir=/var/lib/containers/storage/overlay/b57f721b7589f6c38cc11d76c7fd232e1d8bf7d0a7e499904c4895c0bd8c94fe/diff,workdir=/var/lib/containers/storage/overlay/b57f721b7589f6c38cc11d76c7fd232e1d8bf7d0a7e499904c4895c0bd8c94fe/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/5d8936370cb3f08d888268a755d322e7961886fbf40d9a7986e7615dde95e7be/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/5HLNYNHNTDCLYE5B6US6MW5R5F,upperdir=/var/lib/containers/storage/overlay/5d8936370cb3f08d888268a755d322e7961886fbf40d9a7986e7615dde95e7be/diff,workdir=/var/lib/containers/storage/overlay/5d8936370cb3f08d888268a755d322e7961886fbf40d9a7986e7615dde95e7be/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/97d6e4ac95c851137c98929c3a002a2f60156199900fe069eae7c950ab1d3e58/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/MGEWKUVJJLS4ZNQNDBL7WJS6WR:/var/lib/containers/storage/overlay/l/MKTTXTOG262QMRM3YE64W76H7W:/var/lib/containers/storage/overlay/l/SVERY2GZKEPII2W5SKXLLMSH7S:/var/lib/containers/storage/overlay/l/VREJGHJWRRZFEUUPQJ5S2T6UZX:/var/lib/containers/storage/overlay/l/UBU6THYEGYRN43ZBY5SH4TWHG7:/var/lib/containers/storage/overlay/l/GNBHGN5R7OV6MMVRKDVZPZZNVI:/var/lib/containers/storage/overlay/l/Z3GC37GYDGY55NPPMFBIKBC3JS:/var/lib/containers/storage/overlay/l/PQB3B43EHVCUDSCKTNHOBM4K72:/var/lib/containers/storage/overlay/l/YBYVSIIAXLXCNSYRHZCDFOUB75:/var/lib/containers/storage/overlay/l/UH2ZQMTMOPAREMYC2INGOMH34O:/var/lib/containers/storage/overlay/l/KXAL6NPLPREA337JPCLN4O5KWK:/var/lib/containers/storage/overlay/l/NMSETGCO33RDW73DHQRBU7FSQP,upperdir=/var/lib/containers/storage/overlay/97d6e4ac95c851137c98929c3a002a2f60156199900fe069eae7c950ab1d3e58/diff,workdir=/var/lib/containers/storage/overlay/97d6e4ac95c851137c98929c3a002a2f60156199900fe069eae7c950ab1d3e58/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/7c66a5350e7a6e527f10d2480cdf128458041ef3aced12e59cdbf953e8e29872/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/344W7VQQPULFU5OLHH4H74PRF5:/var/lib/containers/storage/overlay/l/DRR2NEPI4XZI5P5MGC6DNB5IRH,upperdir=/var/lib/containers/storage/overlay/7c66a5350e7a6e527f10d2480cdf128458041ef3aced12e59cdbf953e8e29872/diff,workdir=/var/lib/containers/storage/overlay/7c66a5350e7a6e527f10d2480cdf128458041ef3aced12e59cdbf953e8e29872/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/4f4abbd776fe52f763489157cded5777abd32e497d669374b0f4cb1f4c2f5948/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/344W7VQQPULFU5OLHH4H74PRF5:/var/lib/containers/storage/overlay/l/DRR2NEPI4XZI5P5MGC6DNB5IRH,upperdir=/var/lib/containers/storage/overlay/4f4abbd776fe52f763489157cded5777abd32e497d669374b0f4cb1f4c2f5948/diff,workdir=/var/lib/containers/storage/overlay/4f4abbd776fe52f763489157cded5777abd32e497d669374b0f4cb1f4c2f5948/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/75f836ccf9e3bd1ef1da8cb50efbc827b31793a127dc01c9f3d4eca96ce6d9b8/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/344W7VQQPULFU5OLHH4H74PRF5:/var/lib/containers/storage/overlay/l/DRR2NEPI4XZI5P5MGC6DNB5IRH,upperdir=/var/lib/containers/storage/overlay/75f836ccf9e3bd1ef1da8cb50efbc827b31793a127dc01c9f3d4eca96ce6d9b8/diff,workdir=/var/lib/containers/storage/overlay/75f836ccf9e3bd1ef1da8cb50efbc827b31793a127dc01c9f3d4eca96ce6d9b8/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/19b10fefd01a2f3c078941b758805e03d0a2b25b173ebdfa5e087956f350dc66/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/344W7VQQPULFU5OLHH4H74PRF5:/var/lib/containers/storage/overlay/l/DRR2NEPI4XZI5P5MGC6DNB5IRH,upperdir=/var/lib/containers/storage/overlay/19b10fefd01a2f3c078941b758805e03d0a2b25b173ebdfa5e087956f350dc66/diff,workdir=/var/lib/containers/storage/overlay/19b10fefd01a2f3c078941b758805e03d0a2b25b173ebdfa5e087956f350dc66/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/b3d3e230907a253ea4fa59af11c12b12b61e9d2da70484ab96d51b3b2b1c665b/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/344W7VQQPULFU5OLHH4H74PRF5:/var/lib/containers/storage/overlay/l/DRR2NEPI4XZI5P5MGC6DNB5IRH,upperdir=/var/lib/containers/storage/overlay/b3d3e230907a253ea4fa59af11c12b12b61e9d2da70484ab96d51b3b2b1c665b/diff,workdir=/var/lib/containers/storage/overlay/b3d3e230907a253ea4fa59af11c12b12b61e9d2da70484ab96d51b3b2b1c665b/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/62d9977117d4b4078c847d654a4d2f8a383ea0819122381823b1b9925b849c55/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/344W7VQQPULFU5OLHH4H74PRF5:/var/lib/containers/storage/overlay/l/DRR2NEPI4XZI5P5MGC6DNB5IRH,upperdir=/var/lib/containers/storage/overlay/62d9977117d4b4078c847d654a4d2f8a383ea0819122381823b1b9925b849c55/diff,workdir=/var/lib/containers/storage/overlay/62d9977117d4b4078c847d654a4d2f8a383ea0819122381823b1b9925b849c55/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 2024-05-08T20:34:05.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:34:05.441 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list 2024-05-08T20:34:05.502 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-08T20:34:05.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:34:05.503 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-08T20:34:05.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-08T20:34:05.584 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link delete ceph-brx 2024-05-08T20:34:05.684 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-08T20:34:05.694 INFO:tasks.cephadm:Teardown begin 2024-05-08T20:34:05.694 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-08T20:34:05.723 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-08T20:34:05.761 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-08T20:34:05.761 DEBUG:teuthology.orchestra.run.smithi083:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-08T20:34:05.783 DEBUG:teuthology.orchestra.run.smithi188:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-08T20:34:05.822 INFO:tasks.cephadm:Stopping all daemons... 2024-05-08T20:34:05.822 INFO:tasks.cephadm.mon.smithi083:Stopping mon.smithi083... 2024-05-08T20:34:05.822 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-526038a6-0d75-11ef-bc97-c7b262605968@mon.smithi083 2024-05-08T20:34:06.098 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:34:05 smithi083 systemd[1]: Stopping Ceph mon.smithi083 for 526038a6-0d75-11ef-bc97-c7b262605968... 2024-05-08T20:34:06.099 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:34:06 smithi083 ceph-526038a6-0d75-11ef-bc97-c7b262605968-mon-smithi083[38672]: 2024-05-08T20:34:06.082+0000 7f029f0b6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi083 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-08T20:34:06.099 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:34:06 smithi083 ceph-526038a6-0d75-11ef-bc97-c7b262605968-mon-smithi083[38672]: 2024-05-08T20:34:06.082+0000 7f029f0b6700 -1 mon.smithi083@0(leader) e2 *** Got Signal Terminated *** 2024-05-08T20:34:06.947 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:May 08 20:34:06 smithi083 bash[116462]: ceph-526038a6-0d75-11ef-bc97-c7b262605968-mon-smithi083 2024-05-08T20:34:07.142 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-526038a6-0d75-11ef-bc97-c7b262605968@mon.smithi083.service' 2024-05-08T20:34:07.188 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-08T20:34:07.188 INFO:tasks.cephadm.mon.smithi083:Stopped mon.smithi083 2024-05-08T20:34:07.188 INFO:tasks.cephadm.mon.smithi188:Stopping mon.smithi188... 2024-05-08T20:34:07.188 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-526038a6-0d75-11ef-bc97-c7b262605968@mon.smithi188 2024-05-08T20:34:07.481 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:34:07 smithi188 systemd[1]: Stopping Ceph mon.smithi188 for 526038a6-0d75-11ef-bc97-c7b262605968... 2024-05-08T20:34:07.791 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:34:07 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-mon-smithi188[48733]: 2024-05-08T20:34:07.475+0000 7f91c5140700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi188 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-08T20:34:07.792 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:34:07 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-mon-smithi188[48733]: 2024-05-08T20:34:07.475+0000 7f91c5140700 -1 mon.smithi188@1(peon) e2 *** Got Signal Terminated *** 2024-05-08T20:34:08.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:May 08 20:34:08 smithi188 bash[109742]: ceph-526038a6-0d75-11ef-bc97-c7b262605968-mon-smithi188 2024-05-08T20:34:08.398 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-526038a6-0d75-11ef-bc97-c7b262605968@mon.smithi188.service' 2024-05-08T20:34:08.461 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-08T20:34:08.462 INFO:tasks.cephadm.mon.smithi188:Stopped mon.smithi188 2024-05-08T20:34:08.462 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-08T20:34:08.462 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.0 2024-05-08T20:34:08.745 INFO:journalctl@ceph.osd.0.smithi083.stdout:May 08 20:34:08 smithi083 systemd[1]: Stopping Ceph osd.0 for 526038a6-0d75-11ef-bc97-c7b262605968... 2024-05-08T20:34:09.197 INFO:journalctl@ceph.osd.0.smithi083.stdout:May 08 20:34:08 smithi083 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-0[63164]: 2024-05-08T20:34:08.740+0000 7f47f70be700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-08T20:34:09.197 INFO:journalctl@ceph.osd.0.smithi083.stdout:May 08 20:34:08 smithi083 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-0[63164]: 2024-05-08T20:34:08.740+0000 7f47f70be700 -1 osd.0 72 *** Got signal Terminated *** 2024-05-08T20:34:09.197 INFO:journalctl@ceph.osd.0.smithi083.stdout:May 08 20:34:08 smithi083 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-0[63164]: 2024-05-08T20:34:08.740+0000 7f47f70be700 -1 osd.0 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-08T20:34:14.947 INFO:journalctl@ceph.osd.0.smithi083.stdout:May 08 20:34:14 smithi083 bash[116622]: ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-0 2024-05-08T20:34:16.634 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.0.service' 2024-05-08T20:34:16.680 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-08T20:34:16.680 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-08T20:34:16.680 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-08T20:34:16.680 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.1 2024-05-08T20:34:16.948 INFO:journalctl@ceph.osd.1.smithi083.stdout:May 08 20:34:16 smithi083 systemd[1]: Stopping Ceph osd.1 for 526038a6-0d75-11ef-bc97-c7b262605968... 2024-05-08T20:34:17.447 INFO:journalctl@ceph.osd.1.smithi083.stdout:May 08 20:34:17 smithi083 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-1[71475]: 2024-05-08T20:34:17.008+0000 7f553ec6a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-08T20:34:17.448 INFO:journalctl@ceph.osd.1.smithi083.stdout:May 08 20:34:17 smithi083 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-1[71475]: 2024-05-08T20:34:17.008+0000 7f553ec6a700 -1 osd.1 72 *** Got signal Terminated *** 2024-05-08T20:34:17.448 INFO:journalctl@ceph.osd.1.smithi083.stdout:May 08 20:34:17 smithi083 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-1[71475]: 2024-05-08T20:34:17.008+0000 7f553ec6a700 -1 osd.1 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-08T20:34:23.197 INFO:journalctl@ceph.osd.1.smithi083.stdout:May 08 20:34:22 smithi083 bash[116866]: ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-1 2024-05-08T20:34:24.886 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.1.service' 2024-05-08T20:34:24.927 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-08T20:34:24.928 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-08T20:34:24.928 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-08T20:34:24.928 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.2 2024-05-08T20:34:25.198 INFO:journalctl@ceph.osd.2.smithi083.stdout:May 08 20:34:24 smithi083 systemd[1]: Stopping Ceph osd.2 for 526038a6-0d75-11ef-bc97-c7b262605968... 2024-05-08T20:34:25.697 INFO:journalctl@ceph.osd.2.smithi083.stdout:May 08 20:34:25 smithi083 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-2[79759]: 2024-05-08T20:34:25.223+0000 7ff18490b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-08T20:34:25.697 INFO:journalctl@ceph.osd.2.smithi083.stdout:May 08 20:34:25 smithi083 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-2[79759]: 2024-05-08T20:34:25.223+0000 7ff18490b700 -1 osd.2 72 *** Got signal Terminated *** 2024-05-08T20:34:25.697 INFO:journalctl@ceph.osd.2.smithi083.stdout:May 08 20:34:25 smithi083 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-2[79759]: 2024-05-08T20:34:25.223+0000 7ff18490b700 -1 osd.2 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-08T20:34:31.447 INFO:journalctl@ceph.osd.2.smithi083.stdout:May 08 20:34:31 smithi083 bash[117108]: ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-2 2024-05-08T20:34:33.148 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.2.service' 2024-05-08T20:34:33.190 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-08T20:34:33.190 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-08T20:34:33.190 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-08T20:34:33.191 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.3 2024-05-08T20:34:33.447 INFO:journalctl@ceph.osd.3.smithi083.stdout:May 08 20:34:33 smithi083 systemd[1]: Stopping Ceph osd.3 for 526038a6-0d75-11ef-bc97-c7b262605968... 2024-05-08T20:34:33.947 INFO:journalctl@ceph.osd.3.smithi083.stdout:May 08 20:34:33 smithi083 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-3[89017]: 2024-05-08T20:34:33.503+0000 7fbc2c26d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-08T20:34:33.947 INFO:journalctl@ceph.osd.3.smithi083.stdout:May 08 20:34:33 smithi083 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-3[89017]: 2024-05-08T20:34:33.503+0000 7fbc2c26d700 -1 osd.3 72 *** Got signal Terminated *** 2024-05-08T20:34:33.947 INFO:journalctl@ceph.osd.3.smithi083.stdout:May 08 20:34:33 smithi083 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-3[89017]: 2024-05-08T20:34:33.503+0000 7fbc2c26d700 -1 osd.3 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-08T20:34:35.598 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:35 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:35.342+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:36.847 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:36 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:36.392+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:36.947 INFO:journalctl@ceph.osd.3.smithi083.stdout:May 08 20:34:36 smithi083 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-3[89017]: 2024-05-08T20:34:36.591+0000 7fbc2485b700 -1 osd.3 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.549725+0000 front 2024-05-08T20:34:10.549702+0000 (oldest deadline 2024-05-08T20:34:35.750355+0000) 2024-05-08T20:34:37.847 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:37 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:37.384+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:37.848 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:37 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:37.393+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.001549+0000 front 2024-05-08T20:34:11.001734+0000 (oldest deadline 2024-05-08T20:34:36.901698+0000) 2024-05-08T20:34:37.947 INFO:journalctl@ceph.osd.3.smithi083.stdout:May 08 20:34:37 smithi083 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-3[89017]: 2024-05-08T20:34:37.556+0000 7fbc2485b700 -1 osd.3 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.549725+0000 front 2024-05-08T20:34:10.549702+0000 (oldest deadline 2024-05-08T20:34:35.750355+0000) 2024-05-08T20:34:38.847 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:38 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:38.402+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.001549+0000 front 2024-05-08T20:34:11.001734+0000 (oldest deadline 2024-05-08T20:34:36.901698+0000) 2024-05-08T20:34:38.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:38 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:38.418+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:39.347 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:38 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:38.974+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:34:39.697 INFO:journalctl@ceph.osd.3.smithi083.stdout:May 08 20:34:39 smithi083 bash[117351]: ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-3 2024-05-08T20:34:39.847 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:39 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:39.382+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.001549+0000 front 2024-05-08T20:34:11.001734+0000 (oldest deadline 2024-05-08T20:34:36.901698+0000) 2024-05-08T20:34:39.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:39 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:39.468+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:40.347 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:39 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:39.951+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:34:40.847 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:40 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:40.410+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.001549+0000 front 2024-05-08T20:34:11.001734+0000 (oldest deadline 2024-05-08T20:34:36.901698+0000) 2024-05-08T20:34:40.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:40 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:40.482+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:40.848 INFO:journalctl@ceph.osd.4.smithi188.stdout:May 08 20:34:40 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-4[55489]: 2024-05-08T20:34:40.385+0000 7fea33c17700 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:12.785681+0000 front 2024-05-08T20:34:12.785913+0000 (oldest deadline 2024-05-08T20:34:39.786213+0000) 2024-05-08T20:34:41.246 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.3.service' 2024-05-08T20:34:41.291 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-08T20:34:41.292 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-08T20:34:41.292 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-08T20:34:41.292 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.4 2024-05-08T20:34:41.324 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:40 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:40.913+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:34:41.598 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:41 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:41.380+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.001549+0000 front 2024-05-08T20:34:11.001734+0000 (oldest deadline 2024-05-08T20:34:36.901698+0000) 2024-05-08T20:34:41.598 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:41 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:41.483+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:41.598 INFO:journalctl@ceph.osd.4.smithi188.stdout:May 08 20:34:41 smithi188 systemd[1]: Stopping Ceph osd.4 for 526038a6-0d75-11ef-bc97-c7b262605968... 2024-05-08T20:34:41.598 INFO:journalctl@ceph.osd.4.smithi188.stdout:May 08 20:34:41 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-4[55489]: 2024-05-08T20:34:41.337+0000 7fea33c17700 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:12.785681+0000 front 2024-05-08T20:34:12.785913+0000 (oldest deadline 2024-05-08T20:34:39.786213+0000) 2024-05-08T20:34:41.946 INFO:journalctl@ceph.osd.4.smithi188.stdout:May 08 20:34:41 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-4[55489]: 2024-05-08T20:34:41.604+0000 7fea37621700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-08T20:34:41.946 INFO:journalctl@ceph.osd.4.smithi188.stdout:May 08 20:34:41 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-4[55489]: 2024-05-08T20:34:41.604+0000 7fea37621700 -1 osd.4 72 *** Got signal Terminated *** 2024-05-08T20:34:41.946 INFO:journalctl@ceph.osd.4.smithi188.stdout:May 08 20:34:41 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-4[55489]: 2024-05-08T20:34:41.604+0000 7fea37621700 -1 osd.4 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-08T20:34:42.322 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:41 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:41.941+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:34:42.597 INFO:journalctl@ceph.osd.4.smithi188.stdout:May 08 20:34:42 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-4[55489]: 2024-05-08T20:34:42.318+0000 7fea33c17700 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:12.785681+0000 front 2024-05-08T20:34:12.785913+0000 (oldest deadline 2024-05-08T20:34:39.786213+0000) 2024-05-08T20:34:42.598 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:42 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:42.355+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.001549+0000 front 2024-05-08T20:34:11.001734+0000 (oldest deadline 2024-05-08T20:34:36.901698+0000) 2024-05-08T20:34:42.598 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:42 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:42.465+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:43.320 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:42 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:42.967+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:34:43.597 INFO:journalctl@ceph.osd.4.smithi188.stdout:May 08 20:34:43 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-4[55489]: 2024-05-08T20:34:43.354+0000 7fea33c17700 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:12.785681+0000 front 2024-05-08T20:34:12.785913+0000 (oldest deadline 2024-05-08T20:34:39.786213+0000) 2024-05-08T20:34:43.598 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:43 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:43.466+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:43.598 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:43 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:43.316+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.001549+0000 front 2024-05-08T20:34:11.001734+0000 (oldest deadline 2024-05-08T20:34:36.901698+0000) 2024-05-08T20:34:43.598 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:43 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:43.316+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:18.002870+0000 front 2024-05-08T20:34:18.002935+0000 (oldest deadline 2024-05-08T20:34:42.603435+0000) 2024-05-08T20:34:44.320 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:43 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:43.989+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:34:44.597 INFO:journalctl@ceph.osd.4.smithi188.stdout:May 08 20:34:44 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-4[55489]: 2024-05-08T20:34:44.316+0000 7fea33c17700 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:12.785681+0000 front 2024-05-08T20:34:12.785913+0000 (oldest deadline 2024-05-08T20:34:39.786213+0000) 2024-05-08T20:34:44.598 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:44 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:44.351+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.001549+0000 front 2024-05-08T20:34:11.001734+0000 (oldest deadline 2024-05-08T20:34:36.901698+0000) 2024-05-08T20:34:44.598 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:44 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:44.351+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:18.002870+0000 front 2024-05-08T20:34:18.002935+0000 (oldest deadline 2024-05-08T20:34:42.603435+0000) 2024-05-08T20:34:44.598 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:44 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:44.426+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:45.347 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:45 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:45.018+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:34:45.847 INFO:journalctl@ceph.osd.4.smithi188.stdout:May 08 20:34:45 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-4[55489]: 2024-05-08T20:34:45.356+0000 7fea33c17700 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:12.785681+0000 front 2024-05-08T20:34:12.785913+0000 (oldest deadline 2024-05-08T20:34:39.786213+0000) 2024-05-08T20:34:45.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:45 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:45.454+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:45.848 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:45 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:45.398+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.001549+0000 front 2024-05-08T20:34:11.001734+0000 (oldest deadline 2024-05-08T20:34:36.901698+0000) 2024-05-08T20:34:45.848 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:45 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:45.398+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:18.002870+0000 front 2024-05-08T20:34:18.002935+0000 (oldest deadline 2024-05-08T20:34:42.603435+0000) 2024-05-08T20:34:46.349 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:45 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:45.968+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:34:46.848 INFO:journalctl@ceph.osd.4.smithi188.stdout:May 08 20:34:46 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-4[55489]: 2024-05-08T20:34:46.388+0000 7fea33c17700 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:12.785681+0000 front 2024-05-08T20:34:12.785913+0000 (oldest deadline 2024-05-08T20:34:39.786213+0000) 2024-05-08T20:34:46.848 INFO:journalctl@ceph.osd.4.smithi188.stdout:May 08 20:34:46 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-4[55489]: 2024-05-08T20:34:46.388+0000 7fea33c17700 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:19.786602+0000 front 2024-05-08T20:34:19.786583+0000 (oldest deadline 2024-05-08T20:34:45.686565+0000) 2024-05-08T20:34:46.848 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:46 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:46.409+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.001549+0000 front 2024-05-08T20:34:11.001734+0000 (oldest deadline 2024-05-08T20:34:36.901698+0000) 2024-05-08T20:34:46.848 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:46 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:46.409+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:18.002870+0000 front 2024-05-08T20:34:18.002935+0000 (oldest deadline 2024-05-08T20:34:42.603435+0000) 2024-05-08T20:34:46.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:46 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:46.450+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:47.347 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:46 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:46.948+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:34:47.746 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:47 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:47.480+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:47.746 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:47 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:47.423+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.001549+0000 front 2024-05-08T20:34:11.001734+0000 (oldest deadline 2024-05-08T20:34:36.901698+0000) 2024-05-08T20:34:47.746 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:47 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:47.423+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:18.002870+0000 front 2024-05-08T20:34:18.002935+0000 (oldest deadline 2024-05-08T20:34:42.603435+0000) 2024-05-08T20:34:48.097 INFO:journalctl@ceph.osd.4.smithi188.stdout:May 08 20:34:47 smithi188 bash[109899]: ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-4 2024-05-08T20:34:48.098 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:47 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:47.927+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:34:48.098 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:47 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:47.927+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.535727+0000 front 2024-05-08T20:34:21.535715+0000 (oldest deadline 2024-05-08T20:34:47.435560+0000) 2024-05-08T20:34:48.837 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:48 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:48.377+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.001549+0000 front 2024-05-08T20:34:11.001734+0000 (oldest deadline 2024-05-08T20:34:36.901698+0000) 2024-05-08T20:34:48.838 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:48 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:48.377+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:18.002870+0000 front 2024-05-08T20:34:18.002935+0000 (oldest deadline 2024-05-08T20:34:42.603435+0000) 2024-05-08T20:34:48.838 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:48 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:48.490+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:48.838 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:48 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:48.490+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.938492+0000 front 2024-05-08T20:34:21.938505+0000 (oldest deadline 2024-05-08T20:34:47.838429+0000) 2024-05-08T20:34:49.097 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:48 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:48.882+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:34:49.098 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:48 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:48.882+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.535727+0000 front 2024-05-08T20:34:21.535715+0000 (oldest deadline 2024-05-08T20:34:47.435560+0000) 2024-05-08T20:34:49.443 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:49 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:49.410+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.001549+0000 front 2024-05-08T20:34:11.001734+0000 (oldest deadline 2024-05-08T20:34:36.901698+0000) 2024-05-08T20:34:49.443 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:49 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:49.410+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:18.002870+0000 front 2024-05-08T20:34:18.002935+0000 (oldest deadline 2024-05-08T20:34:42.603435+0000) 2024-05-08T20:34:49.451 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.4.service' 2024-05-08T20:34:49.535 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-08T20:34:49.535 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-08T20:34:49.535 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-08T20:34:49.536 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.5 2024-05-08T20:34:49.848 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:49 smithi188 systemd[1]: Stopping Ceph osd.5 for 526038a6-0d75-11ef-bc97-c7b262605968... 2024-05-08T20:34:49.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:49 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:49.463+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:49.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:49 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:49.463+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.938492+0000 front 2024-05-08T20:34:21.938505+0000 (oldest deadline 2024-05-08T20:34:47.838429+0000) 2024-05-08T20:34:50.190 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:49 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:49.847+0000 7f616a7c6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-08T20:34:50.190 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:49 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:49.847+0000 7f616a7c6700 -1 osd.5 72 *** Got signal Terminated *** 2024-05-08T20:34:50.190 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:49 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:49.847+0000 7f616a7c6700 -1 osd.5 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-08T20:34:50.190 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:49 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:49.914+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:34:50.191 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:49 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:49.914+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.535727+0000 front 2024-05-08T20:34:21.535715+0000 (oldest deadline 2024-05-08T20:34:47.435560+0000) 2024-05-08T20:34:50.476 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:50 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:50.369+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.001549+0000 front 2024-05-08T20:34:11.001734+0000 (oldest deadline 2024-05-08T20:34:36.901698+0000) 2024-05-08T20:34:50.476 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:50 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:50.369+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:18.002870+0000 front 2024-05-08T20:34:18.002935+0000 (oldest deadline 2024-05-08T20:34:42.603435+0000) 2024-05-08T20:34:50.847 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:50 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:50.473+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:50.847 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:50 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:50.473+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.938492+0000 front 2024-05-08T20:34:21.938505+0000 (oldest deadline 2024-05-08T20:34:47.838429+0000) 2024-05-08T20:34:51.347 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:50 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:50.938+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:34:51.348 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:50 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:50.938+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.535727+0000 front 2024-05-08T20:34:21.535715+0000 (oldest deadline 2024-05-08T20:34:47.435560+0000) 2024-05-08T20:34:51.847 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:51 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:51.368+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.001549+0000 front 2024-05-08T20:34:11.001734+0000 (oldest deadline 2024-05-08T20:34:36.901698+0000) 2024-05-08T20:34:51.848 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:51 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:51.368+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:18.002870+0000 front 2024-05-08T20:34:18.002935+0000 (oldest deadline 2024-05-08T20:34:42.603435+0000) 2024-05-08T20:34:51.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:51 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:51.427+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:51.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:51 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:51.427+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.938492+0000 front 2024-05-08T20:34:21.938505+0000 (oldest deadline 2024-05-08T20:34:47.838429+0000) 2024-05-08T20:34:52.347 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:51 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:51.888+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:34:52.347 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:51 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:51.888+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.535727+0000 front 2024-05-08T20:34:21.535715+0000 (oldest deadline 2024-05-08T20:34:47.435560+0000) 2024-05-08T20:34:52.847 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:52 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:52.377+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.001549+0000 front 2024-05-08T20:34:11.001734+0000 (oldest deadline 2024-05-08T20:34:36.901698+0000) 2024-05-08T20:34:52.848 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:52 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:52.377+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:18.002870+0000 front 2024-05-08T20:34:18.002935+0000 (oldest deadline 2024-05-08T20:34:42.603435+0000) 2024-05-08T20:34:52.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:52 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:52.421+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:52.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:52 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:52.421+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.938492+0000 front 2024-05-08T20:34:21.938505+0000 (oldest deadline 2024-05-08T20:34:47.838429+0000) 2024-05-08T20:34:53.347 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:52 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:52.849+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:34:53.348 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:52 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:52.849+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.535727+0000 front 2024-05-08T20:34:21.535715+0000 (oldest deadline 2024-05-08T20:34:47.435560+0000) 2024-05-08T20:34:53.847 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:53 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:53.422+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.001549+0000 front 2024-05-08T20:34:11.001734+0000 (oldest deadline 2024-05-08T20:34:36.901698+0000) 2024-05-08T20:34:53.848 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:53 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:53.422+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:18.002870+0000 front 2024-05-08T20:34:18.002935+0000 (oldest deadline 2024-05-08T20:34:42.603435+0000) 2024-05-08T20:34:53.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:53 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:53.469+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:53.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:53 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:53.469+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.938492+0000 front 2024-05-08T20:34:21.938505+0000 (oldest deadline 2024-05-08T20:34:47.838429+0000) 2024-05-08T20:34:54.347 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:53 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:53.887+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:34:54.347 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:53 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:53.887+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.535727+0000 front 2024-05-08T20:34:21.535715+0000 (oldest deadline 2024-05-08T20:34:47.435560+0000) 2024-05-08T20:34:54.847 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:54 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:54.412+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.001549+0000 front 2024-05-08T20:34:11.001734+0000 (oldest deadline 2024-05-08T20:34:36.901698+0000) 2024-05-08T20:34:54.848 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:54 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:54.412+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:18.002870+0000 front 2024-05-08T20:34:18.002935+0000 (oldest deadline 2024-05-08T20:34:42.603435+0000) 2024-05-08T20:34:54.848 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:54 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5[62931]: 2024-05-08T20:34:54.412+0000 7f6166dbc700 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:27.904640+0000 front 2024-05-08T20:34:27.904607+0000 (oldest deadline 2024-05-08T20:34:53.804550+0000) 2024-05-08T20:34:54.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:54 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:54.456+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:54.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:54 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:54.456+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.938492+0000 front 2024-05-08T20:34:21.938505+0000 (oldest deadline 2024-05-08T20:34:47.838429+0000) 2024-05-08T20:34:54.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:54 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:54.456+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:27.839041+0000 front 2024-05-08T20:34:27.839051+0000 (oldest deadline 2024-05-08T20:34:53.738958+0000) 2024-05-08T20:34:55.228 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:54 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:54.897+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:34:55.228 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:54 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:54.897+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.535727+0000 front 2024-05-08T20:34:21.535715+0000 (oldest deadline 2024-05-08T20:34:47.435560+0000) 2024-05-08T20:34:55.228 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:54 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:54.897+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:29.637252+0000 front 2024-05-08T20:34:29.637377+0000 (oldest deadline 2024-05-08T20:34:54.237792+0000) 2024-05-08T20:34:55.597 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:55 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:55.407+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:55.598 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:55 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:55.407+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.938492+0000 front 2024-05-08T20:34:21.938505+0000 (oldest deadline 2024-05-08T20:34:47.838429+0000) 2024-05-08T20:34:55.598 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:55 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:55.407+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:27.839041+0000 front 2024-05-08T20:34:27.839051+0000 (oldest deadline 2024-05-08T20:34:53.738958+0000) 2024-05-08T20:34:55.938 INFO:journalctl@ceph.osd.5.smithi188.stdout:May 08 20:34:55 smithi188 bash[110137]: ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-5 2024-05-08T20:34:56.347 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:55 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:55.933+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:34:56.347 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:55 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:55.933+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.535727+0000 front 2024-05-08T20:34:21.535715+0000 (oldest deadline 2024-05-08T20:34:47.435560+0000) 2024-05-08T20:34:56.347 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:55 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:55.933+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:29.637252+0000 front 2024-05-08T20:34:29.637377+0000 (oldest deadline 2024-05-08T20:34:54.237792+0000) 2024-05-08T20:34:56.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:56 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:56.428+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:56.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:56 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:56.428+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.938492+0000 front 2024-05-08T20:34:21.938505+0000 (oldest deadline 2024-05-08T20:34:47.838429+0000) 2024-05-08T20:34:56.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:56 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:56.428+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:27.839041+0000 front 2024-05-08T20:34:27.839051+0000 (oldest deadline 2024-05-08T20:34:53.738958+0000) 2024-05-08T20:34:57.168 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:56 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:56.893+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:34:57.168 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:56 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:56.893+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.535727+0000 front 2024-05-08T20:34:21.535715+0000 (oldest deadline 2024-05-08T20:34:47.435560+0000) 2024-05-08T20:34:57.169 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:56 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:56.893+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:29.637252+0000 front 2024-05-08T20:34:29.637377+0000 (oldest deadline 2024-05-08T20:34:54.237792+0000) 2024-05-08T20:34:57.521 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.5.service' 2024-05-08T20:34:57.562 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-08T20:34:57.562 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-08T20:34:57.562 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-08T20:34:57.562 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.6 2024-05-08T20:34:57.845 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:57 smithi188 systemd[1]: Stopping Ceph osd.6 for 526038a6-0d75-11ef-bc97-c7b262605968... 2024-05-08T20:34:57.845 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:57 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:57.458+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:57.845 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:57 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:57.458+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.938492+0000 front 2024-05-08T20:34:21.938505+0000 (oldest deadline 2024-05-08T20:34:47.838429+0000) 2024-05-08T20:34:57.845 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:57 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:57.458+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:27.839041+0000 front 2024-05-08T20:34:27.839051+0000 (oldest deadline 2024-05-08T20:34:53.738958+0000) 2024-05-08T20:34:58.097 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:57 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:57.841+0000 7fd0be18a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-08T20:34:58.097 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:57 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:57.841+0000 7fd0be18a700 -1 osd.6 72 *** Got signal Terminated *** 2024-05-08T20:34:58.098 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:57 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:57.841+0000 7fd0be18a700 -1 osd.6 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-08T20:34:58.098 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:57 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:57.893+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:34:58.098 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:57 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:57.893+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.535727+0000 front 2024-05-08T20:34:21.535715+0000 (oldest deadline 2024-05-08T20:34:47.435560+0000) 2024-05-08T20:34:58.098 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:57 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:57.893+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:29.637252+0000 front 2024-05-08T20:34:29.637377+0000 (oldest deadline 2024-05-08T20:34:54.237792+0000) 2024-05-08T20:34:58.847 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:58 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:58.480+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:58.847 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:58 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:58.480+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.938492+0000 front 2024-05-08T20:34:21.938505+0000 (oldest deadline 2024-05-08T20:34:47.838429+0000) 2024-05-08T20:34:58.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:58 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:58.480+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:27.839041+0000 front 2024-05-08T20:34:27.839051+0000 (oldest deadline 2024-05-08T20:34:53.738958+0000) 2024-05-08T20:34:59.347 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:58 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:58.877+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:34:59.348 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:58 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:58.877+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.535727+0000 front 2024-05-08T20:34:21.535715+0000 (oldest deadline 2024-05-08T20:34:47.435560+0000) 2024-05-08T20:34:59.348 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:58 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:58.877+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:29.637252+0000 front 2024-05-08T20:34:29.637377+0000 (oldest deadline 2024-05-08T20:34:54.237792+0000) 2024-05-08T20:34:59.847 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:59 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:59.505+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:34:59.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:59 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:59.505+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.938492+0000 front 2024-05-08T20:34:21.938505+0000 (oldest deadline 2024-05-08T20:34:47.838429+0000) 2024-05-08T20:34:59.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:34:59 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:34:59.505+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:27.839041+0000 front 2024-05-08T20:34:27.839051+0000 (oldest deadline 2024-05-08T20:34:53.738958+0000) 2024-05-08T20:35:00.306 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:59 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:59.882+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:35:00.306 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:59 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:59.882+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.535727+0000 front 2024-05-08T20:34:21.535715+0000 (oldest deadline 2024-05-08T20:34:47.435560+0000) 2024-05-08T20:35:00.306 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:34:59 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:34:59.882+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:29.637252+0000 front 2024-05-08T20:34:29.637377+0000 (oldest deadline 2024-05-08T20:34:54.237792+0000) 2024-05-08T20:35:00.597 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:00 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:00.539+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:35:00.598 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:00 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:00.539+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.938492+0000 front 2024-05-08T20:34:21.938505+0000 (oldest deadline 2024-05-08T20:34:47.838429+0000) 2024-05-08T20:35:00.598 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:00 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:00.539+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:27.839041+0000 front 2024-05-08T20:34:27.839051+0000 (oldest deadline 2024-05-08T20:34:53.738958+0000) 2024-05-08T20:35:00.598 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:00 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:00.539+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-05-08T20:34:33.739609+0000 front 2024-05-08T20:34:33.739535+0000 (oldest deadline 2024-05-08T20:34:59.639461+0000) 2024-05-08T20:35:01.348 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:35:00 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:35:00.848+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:35:01.348 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:35:00 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:35:00.848+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.535727+0000 front 2024-05-08T20:34:21.535715+0000 (oldest deadline 2024-05-08T20:34:47.435560+0000) 2024-05-08T20:35:01.348 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:35:00 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:35:00.848+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:29.637252+0000 front 2024-05-08T20:34:29.637377+0000 (oldest deadline 2024-05-08T20:34:54.237792+0000) 2024-05-08T20:35:01.825 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:01 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:01.539+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:35:01.825 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:01 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:01.539+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.938492+0000 front 2024-05-08T20:34:21.938505+0000 (oldest deadline 2024-05-08T20:34:47.838429+0000) 2024-05-08T20:35:01.825 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:01 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:01.539+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:27.839041+0000 front 2024-05-08T20:34:27.839051+0000 (oldest deadline 2024-05-08T20:34:53.738958+0000) 2024-05-08T20:35:01.825 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:01 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:01.539+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-05-08T20:34:33.739609+0000 front 2024-05-08T20:34:33.739535+0000 (oldest deadline 2024-05-08T20:34:59.639461+0000) 2024-05-08T20:35:02.097 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:35:01 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:35:01.821+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:11.133635+0000 front 2024-05-08T20:34:11.133499+0000 (oldest deadline 2024-05-08T20:34:38.134007+0000) 2024-05-08T20:35:02.098 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:35:01 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:35:01.821+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.535727+0000 front 2024-05-08T20:34:21.535715+0000 (oldest deadline 2024-05-08T20:34:47.435560+0000) 2024-05-08T20:35:02.098 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:35:01 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6[70295]: 2024-05-08T20:35:01.821+0000 7fd0ba780700 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:29.637252+0000 front 2024-05-08T20:34:29.637377+0000 (oldest deadline 2024-05-08T20:34:54.237792+0000) 2024-05-08T20:35:02.847 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:02 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:02.572+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:35:02.847 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:02 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:02.572+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.938492+0000 front 2024-05-08T20:34:21.938505+0000 (oldest deadline 2024-05-08T20:34:47.838429+0000) 2024-05-08T20:35:02.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:02 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:02.572+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:27.839041+0000 front 2024-05-08T20:34:27.839051+0000 (oldest deadline 2024-05-08T20:34:53.738958+0000) 2024-05-08T20:35:02.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:02 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:02.572+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-05-08T20:34:33.739609+0000 front 2024-05-08T20:34:33.739535+0000 (oldest deadline 2024-05-08T20:34:59.639461+0000) 2024-05-08T20:35:03.830 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:03 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:03.561+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:35:03.830 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:03 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:03.561+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.938492+0000 front 2024-05-08T20:34:21.938505+0000 (oldest deadline 2024-05-08T20:34:47.838429+0000) 2024-05-08T20:35:03.830 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:03 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:03.561+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:27.839041+0000 front 2024-05-08T20:34:27.839051+0000 (oldest deadline 2024-05-08T20:34:53.738958+0000) 2024-05-08T20:35:03.830 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:03 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:03.561+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-05-08T20:34:33.739609+0000 front 2024-05-08T20:34:33.739535+0000 (oldest deadline 2024-05-08T20:34:59.639461+0000) 2024-05-08T20:35:04.097 INFO:journalctl@ceph.osd.6.smithi188.stdout:May 08 20:35:03 smithi188 bash[110380]: ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-6 2024-05-08T20:35:04.847 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:04 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:04.602+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:35:04.847 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:04 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:04.602+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.938492+0000 front 2024-05-08T20:34:21.938505+0000 (oldest deadline 2024-05-08T20:34:47.838429+0000) 2024-05-08T20:35:04.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:04 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:04.602+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:27.839041+0000 front 2024-05-08T20:34:27.839051+0000 (oldest deadline 2024-05-08T20:34:53.738958+0000) 2024-05-08T20:35:04.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:04 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:04.602+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-05-08T20:34:33.739609+0000 front 2024-05-08T20:34:33.739535+0000 (oldest deadline 2024-05-08T20:34:59.639461+0000) 2024-05-08T20:35:05.598 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:05 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:05.561+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:35:05.598 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:05 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:05.561+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.938492+0000 front 2024-05-08T20:34:21.938505+0000 (oldest deadline 2024-05-08T20:34:47.838429+0000) 2024-05-08T20:35:05.598 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:05 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:05.561+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:27.839041+0000 front 2024-05-08T20:34:27.839051+0000 (oldest deadline 2024-05-08T20:34:53.738958+0000) 2024-05-08T20:35:05.598 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:05 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:05.561+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-05-08T20:34:33.739609+0000 front 2024-05-08T20:34:33.739535+0000 (oldest deadline 2024-05-08T20:34:59.639461+0000) 2024-05-08T20:35:05.640 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.6.service' 2024-05-08T20:35:05.681 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-08T20:35:05.681 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-08T20:35:05.681 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-08T20:35:05.681 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.7 2024-05-08T20:35:05.990 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:05 smithi188 systemd[1]: Stopping Ceph osd.7 for 526038a6-0d75-11ef-bc97-c7b262605968... 2024-05-08T20:35:06.347 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:05 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:05.984+0000 7fb8735db700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-08T20:35:06.347 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:05 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:05.984+0000 7fb8735db700 -1 osd.7 72 *** Got signal Terminated *** 2024-05-08T20:35:06.348 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:05 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:05.984+0000 7fb8735db700 -1 osd.7 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-08T20:35:07.097 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:06 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:06.596+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:35:07.098 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:06 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:06.596+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.938492+0000 front 2024-05-08T20:34:21.938505+0000 (oldest deadline 2024-05-08T20:34:47.838429+0000) 2024-05-08T20:35:07.098 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:06 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:06.596+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:27.839041+0000 front 2024-05-08T20:34:27.839051+0000 (oldest deadline 2024-05-08T20:34:53.738958+0000) 2024-05-08T20:35:07.098 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:06 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:06.596+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-05-08T20:34:33.739609+0000 front 2024-05-08T20:34:33.739535+0000 (oldest deadline 2024-05-08T20:34:59.639461+0000) 2024-05-08T20:35:07.847 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:07 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:07.558+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:35:07.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:07 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:07.558+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.938492+0000 front 2024-05-08T20:34:21.938505+0000 (oldest deadline 2024-05-08T20:34:47.838429+0000) 2024-05-08T20:35:07.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:07 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:07.558+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:27.839041+0000 front 2024-05-08T20:34:27.839051+0000 (oldest deadline 2024-05-08T20:34:53.738958+0000) 2024-05-08T20:35:07.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:07 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:07.558+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-05-08T20:34:33.739609+0000 front 2024-05-08T20:34:33.739535+0000 (oldest deadline 2024-05-08T20:34:59.639461+0000) 2024-05-08T20:35:08.847 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:08 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:08.586+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:35:08.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:08 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:08.586+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.938492+0000 front 2024-05-08T20:34:21.938505+0000 (oldest deadline 2024-05-08T20:34:47.838429+0000) 2024-05-08T20:35:08.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:08 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:08.586+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:27.839041+0000 front 2024-05-08T20:34:27.839051+0000 (oldest deadline 2024-05-08T20:34:53.738958+0000) 2024-05-08T20:35:08.848 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:08 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:08.586+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-05-08T20:34:33.739609+0000 front 2024-05-08T20:34:33.739535+0000 (oldest deadline 2024-05-08T20:34:59.639461+0000) 2024-05-08T20:35:10.097 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:09 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:09.636+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:35:10.097 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:09 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:09.636+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.938492+0000 front 2024-05-08T20:34:21.938505+0000 (oldest deadline 2024-05-08T20:34:47.838429+0000) 2024-05-08T20:35:10.097 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:09 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:09.636+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:27.839041+0000 front 2024-05-08T20:34:27.839051+0000 (oldest deadline 2024-05-08T20:34:53.738958+0000) 2024-05-08T20:35:10.098 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:09 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:09.636+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-05-08T20:34:33.739609+0000 front 2024-05-08T20:34:33.739535+0000 (oldest deadline 2024-05-08T20:34:59.639461+0000) 2024-05-08T20:35:11.098 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:10 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:10.657+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6806 osd.0 since back 2024-05-08T20:34:10.836673+0000 front 2024-05-08T20:34:10.836879+0000 (oldest deadline 2024-05-08T20:34:34.936884+0000) 2024-05-08T20:35:11.098 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:10 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:10.657+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6814 osd.1 since back 2024-05-08T20:34:21.938492+0000 front 2024-05-08T20:34:21.938505+0000 (oldest deadline 2024-05-08T20:34:47.838429+0000) 2024-05-08T20:35:11.098 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:10 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:10.657+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6822 osd.2 since back 2024-05-08T20:34:27.839041+0000 front 2024-05-08T20:34:27.839051+0000 (oldest deadline 2024-05-08T20:34:53.738958+0000) 2024-05-08T20:35:11.098 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:10 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:10.657+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.83:6830 osd.3 since back 2024-05-08T20:34:33.739609+0000 front 2024-05-08T20:34:33.739535+0000 (oldest deadline 2024-05-08T20:34:59.639461+0000) 2024-05-08T20:35:11.098 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:10 smithi188 ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7[77642]: 2024-05-08T20:35:10.657+0000 7fb86bbc9700 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.188:6804 osd.4 since back 2024-05-08T20:34:46.441534+0000 front 2024-05-08T20:34:46.441486+0000 (oldest deadline 2024-05-08T20:35:10.541440+0000) 2024-05-08T20:35:12.347 INFO:journalctl@ceph.osd.7.smithi188.stdout:May 08 20:35:11 smithi188 bash[110622]: ceph-526038a6-0d75-11ef-bc97-c7b262605968-osd-7 2024-05-08T20:35:13.751 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-526038a6-0d75-11ef-bc97-c7b262605968@osd.7.service' 2024-05-08T20:35:13.791 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-08T20:35:13.791 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-08T20:35:13.791 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 526038a6-0d75-11ef-bc97-c7b262605968 --force --keep-logs 2024-05-08T20:35:13.970 INFO:teuthology.orchestra.run.smithi083.stdout:Deleting cluster with fsid: 526038a6-0d75-11ef-bc97-c7b262605968 2024-05-08T20:35:38.246 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 526038a6-0d75-11ef-bc97-c7b262605968 --force --keep-logs 2024-05-08T20:35:38.426 INFO:teuthology.orchestra.run.smithi188.stdout:Deleting cluster with fsid: 526038a6-0d75-11ef-bc97-c7b262605968 2024-05-08T20:35:51.703 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-08T20:35:51.740 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-08T20:35:51.771 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-08T20:35:51.773 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/crash to /home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7697965/remote/smithi083/crash 2024-05-08T20:35:51.774 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/crash -- . 2024-05-08T20:35:51.824 INFO:teuthology.orchestra.run.smithi083.stderr:tar: /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/crash: Cannot open: No such file or directory 2024-05-08T20:35:51.824 INFO:teuthology.orchestra.run.smithi083.stderr:tar: Error is not recoverable: exiting now 2024-05-08T20:35:51.825 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/crash to /home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7697965/remote/smithi188/crash 2024-05-08T20:35:51.826 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/crash -- . 2024-05-08T20:35:51.879 INFO:teuthology.orchestra.run.smithi188.stderr:tar: /var/lib/ceph/526038a6-0d75-11ef-bc97-c7b262605968/crash: Cannot open: No such file or directory 2024-05-08T20:35:51.879 INFO:teuthology.orchestra.run.smithi188.stderr:tar: Error is not recoverable: exiting now 2024-05-08T20:35:51.880 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-08T20:35:51.880 DEBUG:teuthology.orchestra.run.smithi083:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/526038a6-0d75-11ef-bc97-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-05-08T20:35:51.909 INFO:teuthology.orchestra.run.smithi083.stderr:grep: /var/log/ceph/526038a6-0d75-11ef-bc97-c7b262605968/ceph.log: No such file or directory 2024-05-08T20:35:51.911 INFO:tasks.cephadm:Compressing logs... 2024-05-08T20:35:51.911 DEBUG:teuthology.orchestra.run.smithi083:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-08T20:35:51.954 DEBUG:teuthology.orchestra.run.smithi188:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-08T20:35:51.980 INFO:teuthology.orchestra.run.smithi083.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-08T20:35:51.980 INFO:teuthology.orchestra.run.smithi188.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-08T20:37:48.347 INFO:tasks.cephadm:Archiving logs... 2024-05-08T20:37:48.347 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7697965/remote/smithi083/log 2024-05-08T20:37:48.348 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-08T20:38:03.059 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7697965/remote/smithi188/log 2024-05-08T20:38:03.060 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-08T20:38:17.470 INFO:tasks.cephadm:Removing cluster... 2024-05-08T20:38:17.470 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 526038a6-0d75-11ef-bc97-c7b262605968 --force 2024-05-08T20:38:17.652 INFO:teuthology.orchestra.run.smithi083.stdout:Deleting cluster with fsid: 526038a6-0d75-11ef-bc97-c7b262605968 2024-05-08T20:38:17.997 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 526038a6-0d75-11ef-bc97-c7b262605968 --force 2024-05-08T20:38:18.178 INFO:teuthology.orchestra.run.smithi188.stdout:Deleting cluster with fsid: 526038a6-0d75-11ef-bc97-c7b262605968 2024-05-08T20:38:18.508 INFO:tasks.cephadm:Removing cephadm ... 2024-05-08T20:38:18.508 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-08T20:38:18.528 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-08T20:38:18.547 INFO:tasks.cephadm:Teardown complete 2024-05-08T20:38:18.547 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-08T20:38:18.558 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-08T20:38:18.558 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-08T20:38:18.571 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-08T20:38:18.666 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-08T20:38:18.676 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-08T20:38:18.676 DEBUG:teuthology.orchestra.run.smithi083:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-08T20:38:18.679 DEBUG:teuthology.orchestra.run.smithi188:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-08T20:38:18.694 INFO:teuthology.orchestra.run.smithi083.stderr:bash: ntpq: command not found 2024-05-08T20:38:18.696 INFO:teuthology.orchestra.run.smithi188.stderr:bash: ntpq: command not found 2024-05-08T20:38:18.698 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-08T20:38:18.698 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-05-08T20:38:18.698 INFO:teuthology.orchestra.run.smithi083.stdout:^* hv01.front.sepia.ceph.com 3 6 377 29 +970us[ +955us] +/- 49ms 2024-05-08T20:38:18.698 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 23 -1382us[-1382us] +/- 60ms 2024-05-08T20:38:18.698 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 25 +283us[ +283us] +/- 74ms 2024-05-08T20:38:18.698 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T20:38:18.701 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-08T20:38:18.701 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-05-08T20:38:18.701 INFO:teuthology.orchestra.run.smithi188.stdout:^* hv01.front.sepia.ceph.com 3 6 377 26 +1000us[+1002us] +/- 49ms 2024-05-08T20:38:18.701 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 25 -1357us[-1357us] +/- 60ms 2024-05-08T20:38:18.701 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 25 +322us[ +322us] +/- 74ms 2024-05-08T20:38:18.701 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T20:38:18.702 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-08T20:38:18.711 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-08T20:38:18.712 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-08T20:38:18.758 DEBUG:teuthology.orchestra.run.smithi083:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-08T20:38:18.828 DEBUG:teuthology.orchestra.run.smithi188:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-08T20:38:18.896 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-08T20:38:18.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T20:38:18.926 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-08T20:38:18.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T20:38:18.955 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-08T20:38:18.966 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-08T20:38:18.980 INFO:teuthology.task.internal:Duration was 2977.275486 seconds 2024-05-08T20:38:18.980 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-08T20:38:19.030 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-08T20:38:19.031 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-08T20:38:19.033 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-08T20:38:19.078 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-08T20:38:19.083 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-08T20:38:19.210 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-08T20:38:19.210 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi083.front.sepia.ceph.com 2024-05-08T20:38:19.211 DEBUG:teuthology.orchestra.run.smithi083:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-08T20:38:19.234 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi188.front.sepia.ceph.com 2024-05-08T20:38:19.235 DEBUG:teuthology.orchestra.run.smithi188:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-08T20:38:19.257 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-08T20:38:19.257 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-08T20:38:19.277 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-08T20:38:19.348 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-08T20:38:19.348 DEBUG:teuthology.orchestra.run.smithi083:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-08T20:38:19.351 DEBUG:teuthology.orchestra.run.smithi188:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-08T20:38:19.460 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-08T20:38:19.469 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-08T20:38:19.470 DEBUG:teuthology.orchestra.run.smithi083:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-08T20:38:19.496 DEBUG:teuthology.orchestra.run.smithi188:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-08T20:38:19.524 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-08T20:38:19.535 DEBUG:teuthology.orchestra.run.smithi083:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-08T20:38:19.540 DEBUG:teuthology.orchestra.run.smithi188:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-08T20:38:19.563 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = core 2024-05-08T20:38:19.591 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = core 2024-05-08T20:38:19.604 DEBUG:teuthology.orchestra.run.smithi083:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-08T20:38:19.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T20:38:19.634 DEBUG:teuthology.orchestra.run.smithi188:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-08T20:38:19.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T20:38:19.661 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-08T20:38:19.674 INFO:teuthology.task.internal:Transferring archived files... 2024-05-08T20:38:19.675 DEBUG:teuthology.misc:Transferring archived files from smithi083:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7697965/remote/smithi083 2024-05-08T20:38:19.675 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-08T20:38:19.719 DEBUG:teuthology.misc:Transferring archived files from smithi188:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-05-08_19:22:21-fs-wip-pdonnell-testing-20240508.150423-reef-distro-default-smithi/7697965/remote/smithi188 2024-05-08T20:38:19.719 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-08T20:38:19.761 INFO:teuthology.task.internal:Removing archive directory... 2024-05-08T20:38:19.762 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-08T20:38:19.764 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-08T20:38:19.813 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-08T20:38:19.824 INFO:teuthology.task.internal:Not uploading archives. 2024-05-08T20:38:19.824 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-08T20:38:19.873 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-08T20:38:19.874 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-08T20:38:19.876 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-08T20:38:19.891 INFO:teuthology.orchestra.run.smithi083.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 8 20:38 /home/ubuntu/cephtest 2024-05-08T20:38:19.893 INFO:teuthology.orchestra.run.smithi188.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 8 20:38 /home/ubuntu/cephtest 2024-05-08T20:38:19.895 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-08T20:38:19.906 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-08T20:38:19.983 INFO:teuthology.run:Summary data: description: fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} duration: 2977.27548623085 flavor: default owner: scheduled_pdonnell@teuthology success: true 2024-05-08T20:38:19.983 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-08T20:38:20.119 INFO:teuthology.run:pass